Discussion on testing coverage for glom

[tonybaloney]

I was thinking about the challenge of calculating the "coverage" of Glom that @mahmoud raised on the Test & Code podcast.

Manually writing parameterised tests for Pytest would be cumbersome and also you wouldn't know the coverage

In the JSON specification, there are objects and arrays, an object can contain values with a fixed list of types, and an array can contain either arrays or objects.

and then an array

As there is only a fixed number of types for a value,

If you converted this into a feature matrix, you could then (deciding on N) first, map out the potential combinations.

Feature 1	Feature 2	Feature 3	Feature 4	Feature-N
null	object	object	object	object
	string	number	array	array
				string

Then converting that feature matrix into a numpy array, you could dynamically generate all of the possible combinations. Since JSON supports an infinite level of nesting, you would have to fix the limted depth to N.

Once you have this you can calculate the possible number of combinations, create test data for each and use them as parameterised values.

Then, since Glom is a DSL, you again decide on N levels of operations-deep and calculate the same feature matrix for Glom.

The possible number of combinations (and your 100% coverage) is then a product of the 2 feature matrices.

You could apply the same technique to generate the same tests.

[kurtbrose]

#66 maybe apt :-)