TestClassOrder does not play nicely with @RepeatedTest

[roded]

Seems like TestClassOrder does not play nicely with @RepeatedTest.

Steps to reproduce

The following results in out of order test execution. I expected all writeTest repetitions to execute before all readTest executions.

Replacing @RepeatedTest with @Test works though.

@TestClassOrder(ClassOrderer.OrderAnnotation.class)
public class RodedIcegenerStressCdsTest {

    @Nested
    @Order(2)
    public class ReadScenarioStressTests {

        // @Test
        @RepeatedTest(10)
        public void readTest() {
            System.out.println("read");
        }
    }

    @Nested
    @Order(1)
    public class WriteScenarioStressTests {

        // @Test
        @RepeatedTest(10)
        public void writeTest() {
            System.out.println("write");
        }
    }
}

Log:

10:04:22.791 INFO  [o.j.j.e.c.EnumConfigurationParameterConverter-ForkJoinPool-1-worker-1] - Using parallel execution mode 'CONCURRENT' set via the 'junit.jupiter.execution.parallel.mode.default' configuration parameter.
read
read
write
write
write
read
read
read
read
read
write
write
write
write
read
write
write
read
write
read

Context

JUnit version: 5.11.3
Running via Gradle on IntelliJ.

    systemProperty("junit.jupiter.execution.parallel.enabled", "true")
    systemProperty("junit.jupiter.execution.parallel.mode.default", "concurrent")
    systemProperty("junit.jupiter.execution.parallel.mode.classes.default", "concurrent")
    systemProperty("junit.jupiter.extensions.autodetection.enabled", "true")
    systemProperty("junit.jupiter.execution.parallel.config.strategy", "fixed")
    systemProperty("junit.jupiter.execution.parallel.config.fixed.parallelism", 10)
    systemProperty("junit.jupiter.execution.parallel.config.fixed.max-pool-size", 10)

[sbrannen]

Hi @roded,

Congratulations on submitting your first issue for JUnit 5! 👍

Replacing @RepeatedTest with @Test works though.

That's actually not true. If you introduce multiple @Test methods in each @Nested test class, you will witness the same behavior that you see with @RepeatedTest.

When you set junit.jupiter.execution.parallel.mode.default = concurrent and junit.jupiter.execution.parallel.mode.classes.default = concurrent, both the test classes and the test methods (or @RepeatedTest invocations) are run currently.

If you want all @Test methods and @RepeatedTest invocations in WriteScenarioStressTests to run first, you should set junit.jupiter.execution.parallel.mode.default = same_thread or simply not configure junit.jupiter.execution.parallel.mode.default (since same_thread is the default).

In light of that, I am closing this issue.

Regards,

Sam

[roded]

@sbrannen Thanks for the reply.
I'm trying to be able to keep the high parallelism plus be able to control the order of the tests.
If I understand correctly, same_thread will limit the parallelism considerably.

[sbrannen]

If you run the test classes in parallel, you will still benefit at that level, which can be significant depending on the number of test classes.

If you only want test methods in certain test classes to execute in the same_thread, you can set the default mode to concurrent and selectively annotate individual test classes with @Execution(SAME_THREAD).

Perhaps @marcphilipp can provide additional insight.

[roded]

I would like to have the opposite: have the nested classes execute one after the other but have their internal @RepeatedTests execute in parallel. Adding @Execution(ExecutionMode.SAME_THREAD) to the parent class and @Execution(ExecutionMode.CONCURRENT) to the nested classes didn't do the trick.

[marcphilipp]

Adding @Execution(ExecutionMode.SAME_THREAD) to the parent class and @Execution(ExecutionMode.CONCURRENT) to the nested classes didn't do the trick.

Have you tried only putting @Execution(ExecutionMode.CONCURRENT) on the methods in the nested class? Alternatively, setting junit.jupiter.execution.parallel.mode.default to concurrent and junit.jupiter.execution.parallel.mode.classes.default to same_thread should achieve that as well (unless you have other test classes where you would want a different behavior).

[roded]

@marcphilipp @Execution(ExecutionMode.CONCURRENT) on the methods works beautifully. Thanks a lot!
JUnit 5 is a joy to work with.

[marcphilipp]

We recently introduced @ResourceLock(target = CHILDREN). I think a similar target attribute for @Execution would be useful for such cases.

@roded Would you mind creating an issue for it in case you agree?

[roded]

@marcphilipp Something here is confusing me.

@Execution(ExecutionMode.SAME_THREAD)
@TestClassOrder(value = ClassOrderer.OrderAnnotation.class)
public class TestClass {

    @Nested
    @Order(1)
    @Execution(ExecutionMode.CONCURRENT)
    public class Nested1 {

        @RepeatedTest(10)
        @Execution(ExecutionMode.CONCURRENT)
        public void test1() {

        }
    }

    @Nested
    @Order(2)
    @Execution(ExecutionMode.CONCURRENT)
    public class Nested2 {

        @RepeatedTest(100)
        @Execution(ExecutionMode.CONCURRENT)
        public void test2() {
        }
    }
}

In the above class as a reference, it makes sense to me that SAME_THREAD on the TestClass means sequential execution of its nested classes (BTW, is this thread a regular test-worker thread which is busy for the entire test duration?).
But why doesn't CONCURRENT on either of the nested classes cause its children (test1x10 and test2x10) run concurrently?
Is it due to RepeatedTest being a node with 10 children itself?

Thanks.

[marcphilipp]

In the above class as a reference, it makes sense to me that SAME_THREAD on the TestClass means sequential execution of its nested classes (BTW, is this thread a regular test-worker thread which is busy for the entire test duration?).

I'm not sure what you mean with "a regular test-worker thread". Declaring SAME_THREAD on a test class means that all class-level lifecycle methods (like @BeforeAll), test methods, and nested classes will be executed on the same thread. Thus, if you omit the @Execution(CONCURRENT) annotations from your example, everything will be executed in the same thread.

But why doesn't CONCURRENT on either of the nested classes cause its children (test1x10 and test2x10) run concurrently?
Is it due to RepeatedTest being a node with 10 children itself?

Do you mean if the test methods in the nested classes wouldn't be annotated with @Execution(CONCURRENT) themselves? If so, it should cause its children to run concurrently. I just tried it myself and what's what happened.