cucumber junit5 engine doesn't support tests rerun

[emaks]

I use cucumber with gradle and run tests with next configuration in build.gradle

dependencies {
    implementation group: 'io.cucumber', name: 'cucumber-junit-platform-engine', version: '7.14.0'
    ...
}
tasks {
    test {
        systemProperties(System.getProperties().sort())
        useJUnitPlatform {
            includeEngines("cucumber")
        }
        systemProperty("cucumber.plugin", "json:build/site/cucumber-report.json, rerun:build/site/failed_scenarios.txt")
        systemProperty("cucumber.features", System.getProperty("folder", "src/test/resources/features"))
        systemProperty("cucumber.glue", "...")
        ...
    }
}

Everting work fine when I just run tests. But if I get "build/site/failed_scenarios.txt" file with failed tests and try to rerun only failed tests with
systemProperty("cucumber.features", "@build/site/failed_scenarios.txt")
I get an error

org.gradle.api.internal.tasks.testing.TestSuiteExecutionException: Could not complete execution for Gradle Test Executor 7.
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:64)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
	at com.sun.proxy.$Proxy2.stop(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker$3.run(TestWorker.java:193)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
	at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
	at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
	at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
Caused by: org.junit.platform.commons.JUnitException: TestEngine with ID 'cucumber' failed to discover tests
	at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.discoverEngineRoot(EngineDiscoveryOrchestrator.java:160)
	at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.discoverSafely(EngineDiscoveryOrchestrator.java:134)
	at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.discover(EngineDiscoveryOrchestrator.java:108)
	at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.discover(EngineDiscoveryOrchestrator.java:80)
	at org.junit.platform.launcher.core.DefaultLauncher.discover(DefaultLauncher.java:110)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
	at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:119)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:94)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:89)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:62)
	... 18 more
Caused by: java.lang.IllegalArgumentException: path must exist: D:\tests\@build\site\failed_scenarios.txt
	at io.cucumber.core.resource.PathScanner.findResourcesForPath(PathScanner.java:49)
	at io.cucumber.core.resource.PathScanner.findResourcesForUri(PathScanner.java:31)
	at io.cucumber.core.resource.ResourceScanner.findResourcesForUri(ResourceScanner.java:61)
	at io.cucumber.core.resource.ResourceScanner.scanForResourcesUri(ResourceScanner.java:134)
	at io.cucumber.junit.platform.engine.FeatureResolver.resolveUri(FeatureResolver.java:231)
	at io.cucumber.junit.platform.engine.FeatureResolver.resolveFeatureWithLines(FeatureResolver.java:246)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
	at io.cucumber.junit.platform.engine.DiscoverySelectorResolver.resolve(DiscoverySelectorResolver.java:71)
	at io.cucumber.junit.platform.engine.DiscoverySelectorResolver.resolveSelectors(DiscoverySelectorResolver.java:48)
	at io.cucumber.junit.platform.engine.CucumberTestEngine.discover(CucumberTestEngine.java:43)
	at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.discoverEngineRoot(EngineDiscoveryOrchestrator.java:152)
	... 28 more

with junit4 I used

task serenityRerun(type: JavaExec) {
    dependsOn(testClasses)
    systemProperties(System.getProperties().sort())
    classpath = sourceSets["test"].runtimeClasspath
    mainClass = 'net.serenitybdd.cucumber.cli.Main'
    args = [
        '--glue', '...',
        '--plugin', '...',
        '@build/site/failed_scenarios.txt'
    ]
}

[emaks]

is there any way to rerun failed tests with junit5?

[mpkorstanje]

To rerun failed scenarios please review https://github.com/cucumber/cucumber-jvm/tree/main/cucumber-junit-platform-engine#rerunning-failed-scenarios.

[mpkorstanje]

While it would be possible to implement a JUnit Engine that executes rerun.txt files, the JUnit 5 Platform does not support ordering test engines. So there is no guarantee that such a cucumber-rerun engine will be invoked after the cucumber engine.

Additionally JUnit 5 somewhat implicitly expects that test execution does not change the tests that could be discovered. While it may be possible with the current state of Surefire and Gradle to ignore that, it could cause all sorts of janky behavior when the JUnit API is used as intended.

More jank than I'm willing to accommodate for.

If this is a major impediment to you, I reckon that you or your company can either petition Gradle and/or Surefire for better JUnit 5 support or materially contribute to that support in some way.

Alternatively, if some one is able to convince the JUnit 5 team to accept a PR that makes it possible order test engines and and willing to commit to dealing with all the jank that comes from implementing cucumber-rerun, I would be willing to accept a PR for that too.

[mpkorstanje]

Additionally JUnit 5 somewhat implicitly expects that test execution does not change the tests that could be discovered.

Potentially implementing TestDescriptor.mayRegisterTests on the rerun engines test descriptor could solve this.

That does leave the test engine ordering problem.

[mpkorstanje]

That does leave the test engine ordering problem.

And the ordering problem could be solved by having multiple surefire executions. One for the run and a second for the rerun. Each targeting different suites. Still janky, but better than nothing.

[NicoIodice]

Just passing by see if there are any advances on this issue.
Isn't still there a solution for this problem where junit reports are not stored for all reruns on a simplier way?

[mpkorstanje]

No, unfortunately the best way to rerun tests and preserve reports written by Cucumber is still to use the JUnit Platform Launcher with different configurations for the run and rerun.

As you are interested in seeing this resolved, would you be interested in contributing a solution?