Executing same suite with different tags wrong behaviour

[JBrexa]

1. Python version python --version - Python 3.10.7
2. Test Junkie version pip show test_junkie - Version: 0.8a6
3. Windows 10
4. Executing a suite:

def order1():

    runner = Runner(suites=[suite_login]
    runner.run( features=
                [
                    Feature.LOGIN,
    
                    ],
        tag_config={"run_on_match_any": 
        [
            Tag.OPERATIONS_LOGIN,
            ]})

def order2():
    runner = Runner(suites=[suite_login,suite]
    runner.run( features=
                [
                    Feature.LOGIN,

                    ],
        tag_config={"run_on_match_any": 
        [
            Tag.OPERATIONS_LOGIN,
            ]})

def order_all():
    funcs = [order1,order2]
    for func in funcs:
        try:
            func()
        except ValueError:
            print('TEST')
            break

order_all()

Expected behavior

I expect that the runner can run the Feature.login twice , 1 after another. This can be very useful if I want to create a suite with different tags and run the same suite multiple times with different tags. This should run my first function then the function 2 separately and so on.

Actual behavior

Right now the first result are correct and real. But the runner only runs the first the suite for the first function, after that only show the results accordingly with the tags. Example of results:
Result of the order1() (Run 1)
========== Test Junkie finished in 4.17 seconds ==========
[4/4 100.00%] SUCCESS
[0/4 0%] FAIL
[0/4 0%] IGNORE
[0/4 0%] ERROR
[0/4 0%] SKIP
[0/4 0%] CANCEL

[SUCCESS] [4/4 100.00%] [3.97s] test_unity_login.suite_login

Result of the order2() (Run 1)
========== Test Junkie finished in 4.17 seconds ==========
[4/4 100.00%] SUCCESS
[0/4 0%] FAIL
[0/4 0%] IGNORE
[0/4 0%] ERROR
[0/4 0%] SKIP
[0/4 0%] CANCEL

[SUCCESS] [4/4 100.00%] [3.97s] test_unity_login.suite_login

It looks good but in reality it only runs the suite one time and I want it to run it twice, one after another. The second result is misleading the runner consider it SUCESS because it use the results from the previous function run.

---

If attaching logs, make sure to remove any personal information, credentials, tokens, keys - anything that you don't
want to become public.

[JBrexa]

Can I exit the runner after run a suite to execute the next one?

[ArturSpirin]

@JBrexa Hey there. I just tried using your code snippet to reproduce the issue but I'm not able to. Granted I did use Python3.9 instead of 3.10. But this is the code that I ran and this is what I got:

from test_junkie.runner import Runner
from tests.junkie_suites.AdvancedSuite import AdvancedSuite
from tests.junkie_suites.BasicSuite import BasicSuite


def order1():
    runner = Runner(suites=[AdvancedSuite])
    runner.run(
        features=["AdvancedSuite"],
        tag_config={
            "run_on_match_any": ["critical2"]
        }
    )


def order2():
    runner = Runner(suites=[AdvancedSuite, BasicSuite])
    runner.run(
        features=["AdvancedSuite"],
        tag_config={
            "run_on_match_any": ["critical2"]
        }
    )


def order_all():
    funcs = [order1, order2]
    for func in funcs:
        try:
            func()
        except ValueError:
            print('TEST')
            break


order_all()

========== Test Junkie finished in 0.24 seconds ==========
[8/14 57.14%] SUCCESS
[1/14 7.14%] FAIL
[0/14 0%] IGNORE
[0/14 0%] ERROR
[5/14 35.71%] SKIP
[0/14 0%] CANCEL

>> [FAIL] [8/14 57.14%] [0.03s] tests.junkie_suites.AdvancedSuite.AdvancedSuite
	|__ test: retry()
	      |__ run #1 [FAIL] [0.00s] :: Traceback: 
		  Traceback (most recent call last):\n  File "C:\\Development\\personal\\test_junkie\\test_junkie\\runner.py", line 479, in __run_test\n    Runner.__process_decorator(decorator_type=DecoratorType.TEST_CASE, suite=suite,\n  File "C:\\Development\\personal\\test_junkie\\test_junkie\\runner.py", line 567, in __process_decorator\n    test.get_function_object()(suite.get_class_instance())\n  File "C:\\Development\\personal\\test_junkie\\tests\\junkie_suites\\AdvancedSuite.py", line 59, in retry\n    assert True is False, "Expected Assertion Error"\nAssertionError: Expected Assertion Error\n
	      |__ run #2 [FAIL] [0.00s] :: Traceback: 
		  Traceback (most recent call last):\n  File "C:\\Development\\personal\\test_junkie\\test_junkie\\runner.py", line 479, in __run_test\n    Runner.__process_decorator(decorator_type=DecoratorType.TEST_CASE, suite=suite,\n  File "C:\\Development\\personal\\test_junkie\\test_junkie\\runner.py", line 567, in __process_decorator\n    test.get_function_object()(suite.get_class_instance())\n  File "C:\\Development\\personal\\test_junkie\\tests\\junkie_suites\\AdvancedSuite.py", line 59, in retry\n    assert True is False, "Expected Assertion Error"\nAssertionError: Expected Assertion Error\n
	      |__ run #3 [FAIL] [0.00s] :: Traceback: 
		  Traceback (most recent call last):\n  File "C:\\Development\\personal\\test_junkie\\test_junkie\\runner.py", line 479, in __run_test\n    Runner.__process_decorator(decorator_type=DecoratorType.TEST_CASE, suite=suite,\n  File "C:\\Development\\personal\\test_junkie\\test_junkie\\runner.py", line 567, in __process_decorator\n    test.get_function_object()(suite.get_class_instance())\n  File "C:\\Development\\personal\\test_junkie\\tests\\junkie_suites\\AdvancedSuite.py", line 59, in retry\n    assert True is False, "Expected Assertion Error"\nAssertionError: Expected Assertion Error\n
	      |__ run #4 [FAIL] [0.00s] :: Traceback: 
		  Traceback (most recent call last):\n  File "C:\\Development\\personal\\test_junkie\\test_junkie\\runner.py", line 479, in __run_test\n    Runner.__process_decorator(decorator_type=DecoratorType.TEST_CASE, suite=suite,\n  File "C:\\Development\\personal\\test_junkie\\test_junkie\\runner.py", line 567, in __process_decorator\n    test.get_function_object()(suite.get_class_instance())\n  File "C:\\Development\\personal\\test_junkie\\tests\\junkie_suites\\AdvancedSuite.py", line 59, in retry\n    assert True is False, "Expected Assertion Error"\nAssertionError: Expected Assertion Error\n

===========================================================
. Test Junkie 0.8a7 (Python3) https://www.test-junkie.com .
===========================================================
========== Test Junkie finished in 0.22 seconds ==========
[12/23 52.17%] SUCCESS
[2/23 8.70%] FAIL
[0/23 0%] IGNORE
[1/23 4.35%] ERROR
[8/23 34.78%] SKIP
[0/23 0%] CANCEL

>> [FAIL] [8/14 57.14%] [0.00s] tests.junkie_suites.AdvancedSuite.AdvancedSuite
	|__ test: retry()
	      |__ run #1 [FAIL] [0.00s] :: Traceback: 
		  Traceback (most recent call last):\n  File "C:\\Development\\personal\\test_junkie\\test_junkie\\runner.py", line 479, in __run_test\n    Runner.__process_decorator(decorator_type=DecoratorType.TEST_CASE, suite=suite,\n  File "C:\\Development\\personal\\test_junkie\\test_junkie\\runner.py", line 567, in __process_decorator\n    test.get_function_object()(suite.get_class_instance())\n  File "C:\\Development\\personal\\test_junkie\\tests\\junkie_suites\\AdvancedSuite.py", line 59, in retry\n    assert True is False, "Expected Assertion Error"\nAssertionError: Expected Assertion Error\n
	      |__ run #2 [FAIL] [0.00s] :: Traceback: 
		  Traceback (most recent call last):\n  File "C:\\Development\\personal\\test_junkie\\test_junkie\\runner.py", line 479, in __run_test\n    Runner.__process_decorator(decorator_type=DecoratorType.TEST_CASE, suite=suite,\n  File "C:\\Development\\personal\\test_junkie\\test_junkie\\runner.py", line 567, in __process_decorator\n    test.get_function_object()(suite.get_class_instance())\n  File "C:\\Development\\personal\\test_junkie\\tests\\junkie_suites\\AdvancedSuite.py", line 59, in retry\n    assert True is False, "Expected Assertion Error"\nAssertionError: Expected Assertion Error\n
	      |__ run #3 [FAIL] [0.00s] :: Traceback: 
		  Traceback (most recent call last):\n  File "C:\\Development\\personal\\test_junkie\\test_junkie\\runner.py", line 479, in __run_test\n    Runner.__process_decorator(decorator_type=DecoratorType.TEST_CASE, suite=suite,\n  File "C:\\Development\\personal\\test_junkie\\test_junkie\\runner.py", line 567, in __process_decorator\n    test.get_function_object()(suite.get_class_instance())\n  File "C:\\Development\\personal\\test_junkie\\tests\\junkie_suites\\AdvancedSuite.py", line 59, in retry\n    assert True is False, "Expected Assertion Error"\nAssertionError: Expected Assertion Error\n
	      |__ run #4 [FAIL] [0.00s] :: Traceback: 
		  Traceback (most recent call last):\n  File "C:\\Development\\personal\\test_junkie\\test_junkie\\runner.py", line 479, in __run_test\n    Runner.__process_decorator(decorator_type=DecoratorType.TEST_CASE, suite=suite,\n  File "C:\\Development\\personal\\test_junkie\\test_junkie\\runner.py", line 567, in __process_decorator\n    test.get_function_object()(suite.get_class_instance())\n  File "C:\\Development\\personal\\test_junkie\\tests\\junkie_suites\\AdvancedSuite.py", line 59, in retry\n    assert True is False, "Expected Assertion Error"\nAssertionError: Expected Assertion Error\n
>> [FAIL] [4/9 44.44%] [0.00s] tests.junkie_suites.BasicSuite.BasicSuite
	|__ test: error()
	      |__ run #1 [ERROR] [0.00s] :: Traceback: 
		  Traceback (most recent call last):\n  File "C:\\Development\\personal\\test_junkie\\test_junkie\\runner.py", line 479, in __run_test\n    Runner.__process_decorator(decorator_type=DecoratorType.TEST_CASE, suite=suite,\n  File "C:\\Development\\personal\\test_junkie\\test_junkie\\runner.py", line 567, in __process_decorator\n    test.get_function_object()(suite.get_class_instance())\n  File "C:\\Development\\personal\\test_junkie\\tests\\junkie_suites\\BasicSuite.py", line 34, in error\n    raise Exception("Exception")\nException: Exception\n
	|__ test: failure()
	      |__ run #1 [FAIL] [0.00s] :: Traceback: 
		  Traceback (most recent call last):\n  File "C:\\Development\\personal\\test_junkie\\test_junkie\\runner.py", line 479, in __run_test\n    Runner.__process_decorator(decorator_type=DecoratorType.TEST_CASE, suite=suite,\n  File "C:\\Development\\personal\\test_junkie\\test_junkie\\runner.py", line 567, in __process_decorator\n    test.get_function_object()(suite.get_class_instance())\n  File "C:\\Development\\personal\\test_junkie\\tests\\junkie_suites\\BasicSuite.py", line 30, in failure\n    assert True is False\nAssertionError\n

===========================================================
. Test Junkie 0.8a7 (Python3) https://www.test-junkie.com .
===========================================================

You could try running that same code, those test suites come with test-junkie package, but you will have to add the features to both of those suites and add the tags to the BasicSuite. If you get a different output aka the same between both of those runs then its likely something does not work well while running on Python 3.10. Unfortunately Python 3.10 does not want to run on my local right now. If you get output like I did or similar (it may vary depending where you put the tags in the BasicSuite) but I've added them to the following tests, just for reference:

   @test(tags=["critical2", "v1"])
    def failure(self):
        assert True is False

    @test(tags=["critical2", "v1"])
    def error(self):
        raise Exception("Exception")

    @test(skip=True)
    def skip(self):
        pass

    @test(meta={"name": "skip_function"},
          skip=skip_function)
    def skip_function(self):
        pass

    @test(parameters=[1, 2, 3, 4], tags=["critical2", "v1"])
    def parameters(self, parameter):
        pass

That would indicate that there is something funky in your test suite configuration.

To answer your other question, you do not need to exit the Runner, each instance of Runner is setup to track its own data so what you are trying to do should work just fine with the code that you have.