Backport current IDM tests to prior spec revision certification packages

[cecille]

This requires a combination of technical and policy work.

There are a few IDM test cases that we should consider backporting to earlier revisions. This work is (mostly) a pre-requisite for using the current TH to certify earlier spec revisions.

To back port a test to an earlier certification revision, you will need to

• make any technical required changes (notes below, not guaranteed to be complete)
• ensure it works with devices using earlier spec revisions. This might entail building example apps off of earlier branches, being aware that many of the example apps, especially on earlier branches, are NOT spec compliant. Will require working with the GRL QA team to verify
• open a CCB (stands for change control board, not sure why) to ask the certification staff to announce the change to the test houses so the new tests can be added.

Let's start with 1.3 and 1.2 and re-assess how the fast track certification is going with upgrades before assessing further backwards.
Some of the tests already appear in the earlier revisions, but may have changes. It might be easier to backport an entire script file at once. Note that some of the script files / functions have been split apart etc. since then, so the picks may not be clean.
branches:

• 1.2: https://github.com/project-chip/connectedhomeip/tree/v1.2-branch
• 1.3: https://github.com/project-chip/connectedhomeip/tree/v1.3-branch

TC_DeviceBasicComposition.py

• - TC-IDM-12.1. This should be a straightforward since this is just a dump of the device attributes. We want this back ported because it will give us a better record of the device capabilities. This is a good first test to understand the entire technical / QA / CCB process.
• TC-IDM-11.1 - this should also be straightforard and has been required since matter 1.0, we were just missing the tests
• remainder - need assess how different master is to each of the branches, but if we could get this entire file backported it would be good.

TC_pics_checker.py
references: https://github.com/CHIP-Specifications/chip-test-plans/blob/master/docs/PICS%20Guidelines.md
please check back once you get to this one, we are hoping to update the documentation for PICS this round, but haven't gotten to it yet.

• this is just a single test, but it's a pretty important one. This verifies that the supplied PICS files match the device under test.There are a few challenges here
• The examples do not have canonical PICS XML references. QA may have some they use for testing, but I'm not sure how far they go back
• The PICS XML files provided as part of the cert package are almost certainly incorrect / incomplete for 1.2. We don't have a great tool right now to determine if the blank PICS XML files are correct. I hacked together something here a day before the test event, but never had a chance to productionize it. PICS checker: Add checks for base xml [DNS - informational only, will re-work] connectedhomeip#32279. It's a start, but not a finish. The PICS XML files will need to be updated (will require a CCB) in order to make this test work.

TC_DeviceComposition.py
Everything in this file relies on having a clean and correct data model scrape of the spec. We have scrapes starting at 1.3, but not before. The device type scrape at 1.3 has not been verified. No scrapes for 1.2 exist. The scrape for 1.2 is unlikely to be clean because the spec did not follow the expected format in all cases. We have two tools to get data model scrapes - https://github.com/csa-data-model/projects is the original tool, but we will soon be moving over to alchemy https://github.com/project-chip/alchemy. I would suggest starting with alchemy and using the DM scraper as a double check. If we need to resort to manual updates to the DM XMLs here, that is acceptable, but getting as much from the scraper as possible (possibly with alchemy updates) would be ideal because manual curation of XML has not traditionally served us well. Once the scrapes are complete, the tests should work as-is, provided they are set up to use the correct DM xmls

• consider also making the DM XML version a parameter to the test
• consider setting this based on the stated spec revision in the basic information cluster

TC_AccessChecker.py

• Once you have a data model scrape for the device composition tests above, I think this one should just work.