Index supplementary files for AnVIL (#5000) #5059

nadove-ucsc · 2023-03-17T02:44:09Z

Connected issues: #5000

Checklist

Author

PR is a draft
Target branch is develop
Name of PR branch matches issues/<GitHub handle of author>/<issue#>-<slug>
PR title references all connected issues
PR title matches¹ that of a connected issue _{or comment in PR explains why they're different}
For each connected issue, there is at least one commit whose title references that issue
PR is connected to all connected issues via ZenHub
PR description links to connected issues
Added partial label to PR _{or this PR completely resolves all connected issues}

¹ when the issue title describes a problem, the corresponding PR
title is Fix: followed by the issue title

Author (reindex, API changes)

Added r tag to commit title _{or this PR does not require reindexing}
Added reindex label to PR _{or this PR does not require reindexing}
Added a (compatible changes) or A (incompatible ones) tag to commit title _{or this PR does not modify the Azul service API}
Added API label to connected issues _{or this PR does not modify the Azul service API}

Author (chains)

This PR is blocked by previous PR in the chain _{or this PR is not chained to another PR}
Added base label to the blocking PR _{or this PR is not chained to another PR}
Added chained label to this PR _{or this PR is not chained to another PR}

Author (upgrading)

Documented upgrading of deployments in UPGRADING.rst _{or this PR does not require upgrading}
Added u tag to commit title _{or this PR does not require upgrading}
Added upgrade label to PR _{or this PR does not require upgrading}

Author (operator tasks)

Added checklist items for additional operator tasks _{or this PR does not require additional tasks}

Author (hotfixes)

Added F tag to main commit title _{or this PR does not include permanent fix for a temporary hotfix}
Reverted the temporary hotfixes for any connected issues _{or the prod branch has no temporary hotfixes for any connected issues}

Author (before every review)

Rebased PR branch on develop, squashed old fixups
Ran make requirements_update _{or this PR does not touch requirements*.txt, common.mk, Makefile and Dockerfile}
Added R tag to commit title _{or this PR does not touch requirements*.txt}
Added reqs label to PR _{or this PR does not touch requirements*.txt}
make integration_test passes in personal deployment _{or this PR does not touch functionality that could break the IT}

Peer reviewer (after requesting changes)

Uncheck the Author (before every review) checklists.

Peer reviewer (after approval)

PR is not a draft
Ticket is in Review requested column
Requested review from primary reviewer
Assigned PR to primary reviewer

Primary reviewer (after requesting changes)

Uncheck the before every review checklists. Update the N reviews label.

Primary reviewer (after approval)

Actually approved the PR
Labeled connected issues as demo or no demo
Commented on connected issues about demo expectations _{or all connected issues are labeled no demo}
Decided if PR can be labeled no sandbox
PR title is appropriate as title of merge commit
N reviews label is accurate
Moved ticket to Approved column
Assigned PR to current operator

Operator (before pushing merge the commit)

Operator (after pushing the merge commit)

¹ When pushing the merge commit is skipped due to the PR being
labelled no sandbox, the next build triggered by a PR whose merge commit is
pushed determines this checklist item.

Operator (reindex)

Deleted unreferenced indices in dev _{or this PR does not remove catalogs or otherwise causes unreferenced indices}
Deleted unreferenced indices in anvildev _{or this PR does not remove catalogs or otherwise causes unreferenced indices}
Started reindex in dev _{or this PR does not require reindexing}
Started reindex in anvildev _{or this PR does not require reindexing}
Checked for and triaged indexing failures in dev _{or this PR does not require reindexing}
Checked for and triaged indexing failures in anvildev _{or this PR does not require reindexing}
Emptied fail queues in dev deployment _{or this PR does not require reindexing}
Emptied fail queues in anvildev deployment _{or this PR does not require reindexing}

Operator

Unassigned PR

Shorthand for review comments

L line is too long
W line wrapping is wrong
Q bad quotes
F other formatting problem

src/azul/indexer/__init__.py

src/azul/types.py

src/azul/indexer/__init__.py

coveralls · 2023-03-18T00:33:22Z

Coverage: 84.62% (-0.2%) from 84.78% when pulling c2ee74e on issues/noah-aviel-dove/5000-index-supp-files-anvil into 48fcfd5 on develop.

codecov · 2023-03-18T00:34:19Z

Codecov Report

Merging #5059 (0b50e33) into develop (48fcfd5) will decrease coverage by 0.04%.
The diff coverage is 70.00%.

❗ Current head 0b50e33 differs from pull request most recent head c2ee74e. Consider uploading reports for the commit c2ee74e to get more accurate results

@@             Coverage Diff             @@
##           develop    #5059      +/-   ##
===========================================
- Coverage    84.75%   84.72%   -0.04%     
===========================================
  Files          149      149              
  Lines        18248    18228      -20     
===========================================
- Hits         15467    15443      -24     
- Misses        2781     2785       +4

Impacted Files	Coverage Δ
src/azul/plugins/repository/canned/__init__.py	`0.00% <0.00%> (ø)`
src/azul/plugins/repository/tdr_anvil/__init__.py	`32.03% <28.57%> (-0.30%)`	⬇️
src/azul/plugins/repository/tdr_hca/__init__.py	`89.95% <92.85%> (ø)`
src/azul/indexer/__init__.py	`100.00% <100.00%> (ø)`
src/azul/plugins/repository/dss/__init__.py	`90.43% <100.00%> (ø)`
src/azul/plugins/repository/tdr.py	`91.20% <100.00%> (ø)`

... and 2 files with indirect coverage changes

src/azul/indexer/__init__.py

nadove-ucsc · 2023-03-24T20:06:46Z

Red status is due to the PR being chained causing false positives for CodeQL. Every commit except for the last should be semantically neutral.

dsotirho-ucsc

Nicely done. Just some nitpick suggestions.

dsotirho-ucsc · 2023-03-24T22:01:21Z

src/azul/plugins/repository/tdr_anvil/__init__.py

-            return None
-        else:
-            return self._parse_drs_path(file_ref)
+    def _columns(self, entity_type: EntityType) -> set[str]:


This seems out of place at the bottom of the file. Could it be put either above or below _convert_column?

dsotirho-ucsc · 2023-03-24T22:11:40Z

src/azul/indexer/__init__.py

@@ -52,6 +49,12 @@ class BundleFQID(SupportsLessThan):
    uuid: BundleUUID
    version: BundleVersion

+    def fqid_json(self) -> MutableJSON:


Suggested change

def fqid_json(self) -> MutableJSON:

def as_json(self) -> MutableJSON:

dsotirho-ucsc · 2023-03-24T22:12:05Z

src/azul/plugins/repository/tdr_anvil/__init__.py

+    def fqid_json(self) -> MutableJSON:
+        result = super().fqid_json()


Suggested change

def fqid_json(self) -> MutableJSON:

result = super().fqid_json()

def as_json(self) -> MutableJSON:

result = super().as_json()

Decided at PL to change shape of the bundle notification so that source can be included in the return value of this method.

dsotirho-ucsc · 2023-03-24T22:12:24Z

src/azul/azulclient.py

-                'bundle_uuid': bundle_fqid.uuid,
-                'bundle_version': bundle_fqid.version
-            },
+            'match': bundle_fqid.fqid_json()


Suggested change

'match': bundle_fqid.fqid_json()

'match': bundle_fqid.as_json()

src/azul/plugins/repository/tdr_anvil/__init__.py

github-actions bot added the orange [process] Done by the Azul team label Mar 17, 2023

nadove-ucsc added the reindex:dev [process] PR requires reindexing dev label Mar 17, 2023

github-advanced-security bot found potential problems Mar 17, 2023

View reviewed changes

src/azul/indexer/__init__.py Fixed Show fixed Hide fixed

nadove-ucsc force-pushed the issues/noah-aviel-dove/5000-index-supp-files-anvil branch 3 times, most recently from 4e0d472 to 6173c7c Compare March 17, 2023 23:48

github-advanced-security bot found potential problems Mar 17, 2023

View reviewed changes

src/azul/indexer/__init__.py Fixed Show fixed Hide fixed

src/azul/types.py Fixed Show fixed Hide fixed

nadove-ucsc force-pushed the issues/noah-aviel-dove/5000-index-supp-files-anvil branch 2 times, most recently from 78b9ce3 to 3388fe9 Compare March 18, 2023 00:07

github-advanced-security bot found potential problems Mar 18, 2023

View reviewed changes

src/azul/indexer/__init__.py Fixed Show fixed Hide fixed

nadove-ucsc force-pushed the issues/noah-aviel-dove/5000-index-supp-files-anvil branch from 150f102 to 14bacb2 Compare March 18, 2023 01:16

github-advanced-security bot found potential problems Mar 18, 2023

View reviewed changes

src/azul/indexer/__init__.py Fixed Show fixed Hide fixed

nadove-ucsc force-pushed the issues/noah-aviel-dove/5000-index-supp-files-anvil branch 5 times, most recently from 47617da to 2deef77 Compare March 18, 2023 06:18

nadove-ucsc added the chained [process] PR needs to based of develop before merging label Mar 24, 2023

nadove-ucsc force-pushed the issues/noah-aviel-dove/5000-index-supp-files-anvil branch from 0002e32 to 99522e5 Compare March 24, 2023 00:10

nadove-ucsc changed the base branch from develop to issues/noah-aviel-dove/5053-it-can-bundle-slow March 24, 2023 00:13

nadove-ucsc force-pushed the issues/noah-aviel-dove/5000-index-supp-files-anvil branch 2 times, most recently from ec7ab38 to 0550d85 Compare March 24, 2023 19:45

nadove-ucsc requested a review from dsotirho-ucsc March 24, 2023 20:05

nadove-ucsc assigned dsotirho-ucsc Mar 24, 2023

dsotirho-ucsc requested changes Mar 24, 2023

View reviewed changes

dsotirho-ucsc removed their assignment Mar 24, 2023

hannes-ucsc force-pushed the issues/noah-aviel-dove/5053-it-can-bundle-slow branch from c7fcd9f to 06584fb Compare March 26, 2023 18:35

hannes-ucsc added 3 reviews [process] Lead requested changes thrice and removed 2 reviews [process] Lead requested changes twice labels May 4, 2023

hannes-ucsc removed their assignment May 4, 2023

nadove-ucsc force-pushed the issues/noah-aviel-dove/5000-index-supp-files-anvil branch from 650fc5a to c739bee Compare May 4, 2023 01:04

nadove-ucsc requested a review from hannes-ucsc May 4, 2023 01:52

nadove-ucsc assigned hannes-ucsc May 4, 2023

hannes-ucsc approved these changes May 5, 2023

View reviewed changes

hannes-ucsc assigned achave11-ucsc and unassigned hannes-ucsc May 5, 2023

nadove-ucsc and others added 13 commits May 5, 2023 08:57

Fix type warning

fe36e54

Reorder classes; narrow return type

e6d6a9b

Fix type warnings

4744cc4

Refactor AnVIL plugin column processing

7978d7c

Remove redundant parameter

fcdaf71

Consolidate Generic[...] parameter introspection

ed4ff93

Add TypedDict class for SourceRef

54fea5b

Restructure bundle notifications

fbd9277

Add FIXME (#5171)

1048a27

Add FIXME (#5105)

006a76c

Parameterize repository plugin with bundle FQID class

6865f07

Include entity type in AnVIL bundle FQIDs

095c436

[a r] Index supplementary files for AnVIL (#5000)

c2ee74e

achave11-ucsc force-pushed the issues/noah-aviel-dove/5000-index-supp-files-anvil branch from c739bee to c2ee74e Compare May 5, 2023 15:58

achave11-ucsc added the sandbox [process] Resolution is being verified in sandbox deployment label May 5, 2023

achave11-ucsc merged commit e06862b into develop May 5, 2023

achave11-ucsc deleted the issues/noah-aviel-dove/5000-index-supp-files-anvil branch May 5, 2023 18:15

achave11-ucsc removed their assignment May 5, 2023

dsotirho-ucsc removed the base [process] Another PR needs to be rebased before merging this one label May 8, 2023

dsotirho-ucsc mentioned this pull request May 11, 2023

KeyError: 'is_supplementary' in azul-service-anvildev #5202

Closed

14 tasks

nadove-ucsc mentioned this pull request May 12, 2023

Partition sizing ignores supplementary bundles #5207

Closed

14 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Index supplementary files for AnVIL (#5000) #5059

Index supplementary files for AnVIL (#5000) #5059

nadove-ucsc commented Mar 17, 2023 •

edited by achave11-ucsc

Loading

coveralls commented Mar 18, 2023 •

edited

Loading

codecov bot commented Mar 18, 2023 •

edited

Loading

nadove-ucsc commented Mar 24, 2023

dsotirho-ucsc left a comment

dsotirho-ucsc Mar 24, 2023

dsotirho-ucsc Mar 24, 2023

dsotirho-ucsc Mar 24, 2023

nadove-ucsc Mar 27, 2023

dsotirho-ucsc Mar 24, 2023

	def fqid_json(self) -> MutableJSON:
	def as_json(self) -> MutableJSON:

		def fqid_json(self) -> MutableJSON:
		result = super().fqid_json()

	'match': bundle_fqid.fqid_json()
	'match': bundle_fqid.as_json()

Index supplementary files for AnVIL (#5000) #5059

Index supplementary files for AnVIL (#5000) #5059

Conversation

nadove-ucsc commented Mar 17, 2023 • edited by achave11-ucsc Loading

Checklist

Author

Author (reindex, API changes)

Author (chains)

Author (upgrading)

Author (operator tasks)

Author (hotfixes)

Author (before every review)

Peer reviewer (after requesting changes)

Peer reviewer (after approval)

Primary reviewer (after requesting changes)

Primary reviewer (after approval)

Operator (before pushing merge the commit)

Operator (after pushing the merge commit)

Operator (reindex)

Operator

Shorthand for review comments

coveralls commented Mar 18, 2023 • edited Loading

codecov bot commented Mar 18, 2023 • edited Loading

Codecov Report

nadove-ucsc commented Mar 24, 2023

dsotirho-ucsc left a comment

Choose a reason for hiding this comment

dsotirho-ucsc Mar 24, 2023

Choose a reason for hiding this comment

dsotirho-ucsc Mar 24, 2023

Choose a reason for hiding this comment

dsotirho-ucsc Mar 24, 2023

Choose a reason for hiding this comment

nadove-ucsc Mar 27, 2023

Choose a reason for hiding this comment

dsotirho-ucsc Mar 24, 2023

Choose a reason for hiding this comment

nadove-ucsc commented Mar 17, 2023 •

edited by achave11-ucsc

Loading

coveralls commented Mar 18, 2023 •

edited

Loading

codecov bot commented Mar 18, 2023 •

edited

Loading