Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support streaming vs independent network/tar actions. #1887

Open
gdborton opened this issue Dec 3, 2024 · 0 comments
Open

Support streaming vs independent network/tar actions. #1887

gdborton opened this issue Dec 3, 2024 · 0 comments
Labels
cache enhancement New feature or request

Comments

@gdborton
Copy link

gdborton commented Dec 3, 2024

Describe the enhancement
Hello! I'm working on optimizing some of our CI workloads. One of the things that I noticed in that process is that the actions/cache logic runs download/upload and compress/decompress as separate actions. I think we can squeeze more performance out by utilizing streams and skipping writing to and reading from the fs for the cache.

For the use case I'm looking at specifically, the cache save step takes ~38s and the restore takes 22s (~4s download, ~18s uncompress). I think the biggest benefit for us will come from upload, but I can't tell from the logs how long each portion takes.

Is this something you'd be open to? I am open to help make this happen, but I wanna get confirmation before digging any further.

Code Snippet
Before:

await download(url, location); // <-- downloads, saves to disk
await untar(location); // <-- reads from disk, doesn't start until download is complete

After:

await downloadAndUntar(url); // <-- streams, avoids writing to then reading from fs, starts decompressing immediately

Additional information
Add any other context about the feature here.

@gdborton gdborton added the enhancement New feature or request label Dec 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cache enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants