Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Slow post-action after v4 #878

Closed
2 of 5 tasks
TrygveUrdahl opened this issue Oct 24, 2023 · 23 comments · May be fixed by #907
Closed
2 of 5 tasks

Slow post-action after v4 #878

TrygveUrdahl opened this issue Oct 24, 2023 · 23 comments · May be fixed by #907
Assignees
Labels
bug Something isn't working

Comments

@TrygveUrdahl
Copy link

TrygveUrdahl commented Oct 24, 2023

Description:
The post-action step for this action seems to have become much slower after version 4.0.0 was released.
In the two included images, the only major difference should be the version of actions/setup-node.

Before:
image

After:
image

Action version:
Using actions/setup-node@v4 15 hours after 4.0.0 was released.

Platform:

  • Ubuntu
  • macOS
  • Windows

Runner type:

  • Hosted
  • Self-hosted

Tools version:
Node 18.18.2, and [email protected].

@TrygveUrdahl TrygveUrdahl added bug Something isn't working needs triage labels Oct 24, 2023
@TrygveUrdahl
Copy link
Author

I noticed that my workflow files didn't specify what node version to use for this workflow through any node-version or node-version-file. By adding that, and upgrading my project to use node 20, this issue seems to go away.

@nikolai-laevskii
Copy link
Contributor

Thank you for detailed description of the issue! We'll investigate it and come back with details.

@panva
Copy link
Contributor

panva commented Oct 25, 2023

I noticed that my workflow files didn't specify what node version to use for this workflow through any node-version or node-version-file. By adding that, and upgrading my project to use node 20, this issue seems to go away.

My workflows do always specify node-version but the post-action is slow as well.

I also have a flow log here which despite requesting lts/iron resolved 20.8.1 which isn't lts/iron. This particular setup-node use is from a reused worflow, when setup-node is used outside of a reused workflow the correct 20.9.0 is resolved.

@bodinsamuel
Copy link

@ledermann
Copy link

Same here in multiple projects using JS with Yarn. The Post Setup Node.js job slows down from a few seconds to about 2:20 minutes after upgrading to v4.

It seems to me that this only happens when JS dependencies have been updated. Re-running the job (which causes a cache hit) is fast again - perhaps because the cache doesn't need to be saved.

My conclusion: Saving the cache is significantly slower with v4 than with v3.

@ari-becker
Copy link

Noticing this as well because I have an aggressive timeout-minutes: 1 on my actions/setup-node step, which is now causing many of my builds to fail.

I concur with @ledermann that the issue is somewhere in cache-saving.

With a cache hit, the post-action takes 0s as expected:

image

Without a cache hit, when trying to save the cache, the post-action times out:

image

@eregon
Copy link

eregon commented Nov 7, 2023

See ruby/setup-ruby#543 which explains a possible cause and fix.

@jraoult
Copy link

jraoult commented Nov 8, 2023

And here's the fix: ruby/setup-ruby@036ef45 that should probably be ported to setup-node

sitek94 added a commit to sitek94/pnpm-monorepo that referenced this issue Nov 26, 2023
sitek94 added a commit to sitek94/pnpm-monorepo that referenced this issue Nov 26, 2023
sitek94 added a commit to sitek94/pnpm-monorepo that referenced this issue Nov 26, 2023
* revert setup-node action to v3 because of slow post action step

see issue:
actions/setup-node#878

* fix build commands
xiniria added a commit to xiniria/setup-node that referenced this issue Nov 27, 2023
xiniria added a commit to xiniria/setup-node that referenced this issue Nov 27, 2023
@xiniria
Copy link

xiniria commented Nov 27, 2023

@nikolai-laevskii I opened a PR in #907 that applies the same fix as https://github.com/ruby/setup-ruby (issue ruby/setup-ruby#543, PR ruby/setup-ruby#546), could you take a look at it?

@nrutman
Copy link

nrutman commented Dec 1, 2023

I'm also experiencing this on Node 18.

@xiniria
Copy link

xiniria commented Dec 4, 2023

I'm also experiencing this on Node 18.

@nrutman Actually this issue does not depend on the Node version you're using, but rather the Node version that the GitHub Action runner uses itself (in this case, the runner for @actions/http-client, used by @actions/cache, itself used by @actions/setup-node). We have no control over this other than downgrading to v3 of the action.

@dmitry-shibanov
Copy link
Contributor

Hello everyone. We've merged the fix for the issue. Could you please try to use actions/setup-node@main to confirm that it works as expected?

uncenter added a commit to uncenter/uncenter.dev that referenced this issue Dec 13, 2023
@uncenter
Copy link

Made a typo in that commit 😅 . I'll let you know if it appears to be working or not. It doesn't always have the issue but I'll try to run the action a few times to make sure.
Screenshot 2023-12-13 at 06 48 17 (Arc)

@TrygveUrdahl
Copy link
Author

Hello everyone. We've merged the fix for the issue. Could you please try to use actions/setup-node@main to confirm that it works as expected?

Hi! Yeah, from some quick tests it seems to work better now.

colinrotherham added a commit to alphagov/govuk-frontend that referenced this issue Dec 14, 2023
@colinrotherham
Copy link

@dmitry-shibanov Just confirming that the fix has worked for us too

We only really noticed it on Windows where all those tiny node_modules really affect performance:

Before

Windows cache restore time before (5m 38s)

After

Windows cache restore time after (1m 46s)

@kireerik
Copy link

@dmitry-shibanov Can you release the changes?

@aparnajyothi-y
Copy link
Contributor

Hello everyone. We released the changes yesterday.

For now I am going to close the issue. If you have any concerns feel free to ping us.

@r-dev-limited
Copy link

r-dev-limited commented Feb 3, 2024

  • I am using v4 with node 20. And it basically hangs forever, any idea how to fix it ?

Issue was related to cmd failing, and for some reason NODE action hanged...once I fixed other issue, it works again. SRY to bother

@domhhv
Copy link

domhhv commented Nov 3, 2024

Hi all. I think I am still affected by this issue.

In my app, the "Post Cache node_modules" step takes quite long for some reason (2m on average). As a result, if I install or remove a yarn dependency, the setup job alone takes around 3 minutes before any other job can start, while if I don't change any deps, it takes only 10 seconds, which is great and expected.

But I would still like not to slow the process down when deps change. Is the behavior I described expected and is there anything I can do? I already changed actions/setup-node@v4 to actions/setup-node@main in my GH workflow.

Here's a link to one of the slow setup runs: https://github.com/domhhv/habitrack/actions/runs/11651863472/job/32442340624.

@aparnajyothi-y
Copy link
Contributor

Hello @domhhv, Thanks for sharing the details. The slow ""Post Cache node_modules"" step when dependencies change is somewhat expected, as it involves cache validation or recreation. This behavior is aligned with the caching mechanism used by actions/setup-node, which caches the global package cache (e.g., ~/.npm for npm, ~/.pnpm-store for pnpm), but does not directly cache the node_modules directory to prevent inconsistencies across different environments.

Here are a few suggestions to optimize the process:

Cache Key: Ensure your cache key includes both package.json and yarn.lock to invalidate the cache only when dependencies change. This minimizes unnecessary cache misses and speeds up subsequent jobs:

key: ${{ runner.os }}-yarn-${{ hashFiles('/package.json', '/yarn.lock') }}

Use yarn install --frozen-lockfile: This ensures that Yarn installs dependencies exactly as specified in yarn.lock, which can help prevent unnecessary updates and speed up the process:

  • run: yarn install --frozen-lockfile

Cache Yarn Cache: In addition to node_modules,

consider caching the Yarn cache directory (~/.cache/yarn) to reduce the time spent downloading dependencies from scratch:

  • name: Cache Yarn dependencies
    uses: actions/cache@v4
    with:
    path: ~/.cache/yarn
    key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
    These optimizations should help reduce the setup time when dependencies change.
    Please let me know if you need any further assistance!

@compojoom
Copy link

Hey guys! I wanted to share my experience here. Maybe it could help someone.
I was pulling my hair today and wondered why actions/cache is sooooo slow! Then I found this doc:
https://github.com/actions/cache/blob/main/caching-strategies.md
Which made me think that it could be the calculation of my cache key that is so slow.
You see nextjs is suggesting you do the following:
https://nextjs.org/docs/pages/building-your-application/deploying/ci-build-caching

uses: actions/cache@v4
with:
  # See here for caching with `yarn` https://github.com/actions/cache/blob/main/examples.md#node---yarn or you can leverage caching with actions/setup-node https://github.com/actions/setup-node
  path: |
    ~/.npm
    ${{ github.workspace }}/.next/cache
  # Generate a new cache whenever packages or source files change.
  key: ${{ runner.os }}-nextjs-${{ hashFiles('**/package-lock.json') }}-${{ hashFiles('**/*.js', '**/*.jsx', '**/*.ts', '**/*.tsx') }}
  # If source files changed but packages didn't, rebuild from a prior cache.
  restore-keys: |
    ${{ runner.os }}-nextjs-${{ hashFiles('**/package-lock.json') }}-

As you can notice the hashFiles function is looking for js, jsx, ts and tsx files. In my project we are talking about 1000s of files. So I split the job in restore and save parts and when saving I reuse the key from the restore job.

This actually seems to have fixed the post action for me.

@simon-abbott
Copy link

@compojoom Good thought, but unfortunately that's not the case here. The cache key is calculated during the cache restore step (using only the lockfile) and is re-used without recalculation during the post-job action.

I think this step really is just slow because the cache directory is huge, and thus takes a while to sync across to the cache.

@compojoom
Copy link

@simon-abbott oh sorry. I posted in the wrong repo. setup-node is fast in my jobs. I was having issues with the "actions/cache" job

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.