Partition sizing ignores supplementary bundles

[nadove-ucsc]

Subgraph numbers need to be updated for anvildev and anvilprod following the addition of supplementary bundles. This was overlooked during #5059. Additionally, the list_partitions method for the AnVIL plugin doesn't count supplementary bundles. This may be why we didn't see this error on anvildev, because the mistakes cancelled each other out.

[edit: @dsotirho-ucsc]
@hannes-ucsc: "Lets also add a script that facilitates calculating bundle counts that we hard code for each source in AZUL_CATALOGS."

[edit: @hannes-ucsc] To reproduce, one would run IT against a catalog with 1000G in it. See https://gitlab.prod.anvil.gi.ucsc.edu/ucsc/azul/-/jobs/3859 for example.

This gives

======================================================================
FAIL: test_indexing (integration_test.IndexingIntegrationTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builds/ucsc/azul/test/azul_test_case.py", line 208, in wrapped
    return method(*args, **kwargs)
  File "/builds/ucsc/azul/test/integration_test.py", line 442, in test_indexing
    notifications, fqids = self._prepare_notifications(catalog)
  File "/builds/ucsc/azul/test/integration_test.py", line 928, in _prepare_notifications
    list(starmap(update, self._list_bundles(catalog,
  File "/builds/ucsc/azul/test/integration_test.py", line 339, in _list_bundles
    source, prefix, new_fqids = self._list_partition_bundles(catalog, source)
  File "/builds/ucsc/azul/test/integration_test.py", line 312, in _list_partition_bundles
    self.assertLessEqual(bundle_count, upper, partition + ' is too large')
AssertionError: 1484 not less than or equal to 1024 : Partition '1' of source tdr:datarepo-dev-43738c90:snapshot/ANVIL_1000G_2019_Dev_20230302_ANV5_202303032342:/1 is too large

---

• Security design review completed; the Resolution of this issue does not …
  • … affect authentication; for example:
    • OAuth 2.0 with the application (API or Swagger UI)
    • Authentication of developers with Google Cloud APIs
    • Authentication of developers with AWS APIs
    • Authentication with a GitLab instance in the system
    • Password and 2FA authentication with GitHub
    • API access token authentication with GitHub
    • Authentication with
  • … affect the permissions of internal users like access to
    • Cloud resources on AWS and GCP
    • GitLab repositories, projects and groups, administration
    • an EC2 instance via SSH
    • GitHub issues, pull requests, commits, commit statuses, wikis, repositories, organizations
  • … affect the permissions of external users like access to
    • TDR snapshots
  • … affect permissions of service or bot accounts
    • Cloud resources on AWS and GCP
  • … affect audit logging in the system, like
    • adding, removing or changing a log message that represents an auditable event
    • changing the routing of log messages through the system
  • … affect monitoring of the system
  • … introduce a new software dependency like
    • Python packages on PYPI
    • Command-line utilities
    • Docker images
    • Terraform providers
  • … add an interface that exposes sensitive or confidential data at the security boundary
  • … affect the encryption of data at rest
  • … require persistence of sensitive or confidential data that might require encryption at rest
  • … require unencrypted transmission of data within the security boundary
  • … affect the network security layer; for example by
    • modifying, adding or removing firewall rules
    • modifying, adding or removing security groups
    • changing or adding a port a service, proxy or load balancer listens on
• Documentation on any unchecked boxes is provided in comments below

[hannes-ucsc]

For demo, show passing IT build in anvilprod on develop branch after PR #5184 lands on develop.