Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AWS ECR login fails in windows-2022 20241211.1.0 #11172

Open
2 of 15 tasks
PatTheSilent opened this issue Dec 12, 2024 · 7 comments
Open
2 of 15 tasks

AWS ECR login fails in windows-2022 20241211.1.0 #11172

PatTheSilent opened this issue Dec 12, 2024 · 7 comments
Assignees
Labels
awaiting-deployment Code complete; awaiting deployment and/or deployment in progress bug report OS: Windows

Comments

@PatTheSilent
Copy link

Description

We've started experiencing failures to login to AWS ECR starting from windows-2022 20241211.1.0
We're using "big" runners, which have received the update to 20241211.1.0 on which we're experiencing this failure.
We're not experiencing this on windows-2022 runners, which still have 20241201.2.0

Platforms affected

  • Azure DevOps
  • GitHub Actions - Standard Runners
  • GitHub Actions - Larger Runners

Runner images affected

  • Ubuntu 20.04
  • Ubuntu 22.04
  • Ubuntu 24.04
  • macOS 12
  • macOS 13
  • macOS 13 Arm64
  • macOS 14
  • macOS 14 Arm64
  • macOS 15
  • macOS 15 Arm64
  • Windows Server 2019
  • Windows Server 2022

Image version and build link

Works as expected in 20241201.2.0
fails in 20241211.1.0

Is it regression?

20241201.2.0

Expected behavior

I'm able to authenticate with AWS ECR using aws-actions/amazon-ecr-login@v1

Actual behavior

Error: Could not login to registry redacted.dkr.ecr.eu-west-1.amazonaws.com: WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error saving credentials: error storing credentials - err: exit status 1, out: `A specified logon session does not exist. It may already have been terminated.`

Repro steps

  1. Have the following steps in your workflow:
- name: Configure AWS Credentials
  uses: aws-actions/configure-aws-credentials@v4
  with:
    aws-region: eu-west-1
    role-to-assume: arn:aws:iam::redacted:role/redacted
    role-skip-session-tagging: true
    role-duration-seconds: 3600

- name: Login to Amazon ECR
  id: login-ecr
  uses: aws-actions/amazon-ecr-login@v1
  with:
    mask-password: true
  1. Observe failure
@kishorekumar-anchala
Copy link
Contributor

Hi @PatTheSilent ,

Thank you for raising the issue , we will look into the issue and update Asap

@mavimo
Copy link

mavimo commented Dec 12, 2024

We have a similar issue moving from 20241202.1.0 to 20241211.2.0 where executing docker login fail to save credentials:

Run docker/login-action@v3
Logging into us-docker.pkg.dev/XXXXXXXXX...
Error: Error saving credentials: error storing credentials - err: exit status 1, out: `The stub received bad data.`

@vidyasagarnimmagaddi vidyasagarnimmagaddi added the awaiting-deployment Code complete; awaiting deployment and/or deployment in progress label Dec 16, 2024
@austindrenski
Copy link

Hi folks, just adding that my organization has also run into this issue. Our logs indicate we first experienced the failure at the end of last week (went unnoticed going into the weekend), and we continue to see it this morning.

As with @PatTheSilent, we are encountering this in a workflow that uses a GitHub-hosted large runner pool, and as with @mavimo, we're seeing it during the first docker/[email protected] step in our workflow.

One difference from @mavimo is that we are seeing this failure in 20241215.1.0 (i.e. 2024-12-15).

@kishorekumar-anchala please see below for some lightly sanitized logs, and please let me know if I can provide any additional information to help your investigation. This is a major blocker for building Windows-based Docker images in CI, so hoping our logs may help things along:

2024-12-16T16:40:46.7689783Z ##[debug]Starting: *** / ***
2024-12-16T16:40:46.7710528Z ##[debug]Cleaning runner temp folder: C:\a\_temp
2024-12-16T16:40:46.7811255Z ##[debug]Starting: Set up job
2024-12-16T16:40:46.7812545Z Current runner version: '2.321.0'
2024-12-16T16:40:46.7815921Z Runner name: 'windows-latest-16-cores_b1e07f54e0dd'
2024-12-16T16:40:46.7816683Z Runner group name: 'Default Larger Runners'
2024-12-16T16:40:46.7817474Z Machine name: 'runner'
2024-12-16T16:40:46.7833268Z ##[group]Operating System
2024-12-16T16:40:46.7833912Z Microsoft Windows Server 2022
2024-12-16T16:40:46.7834431Z 10.0.20348
2024-12-16T16:40:46.7834865Z Datacenter
2024-12-16T16:40:46.7835292Z ##[endgroup]
2024-12-16T16:40:46.7835714Z ##[group]Runner Image
2024-12-16T16:40:46.7836219Z Image: windows-2022
2024-12-16T16:40:46.7836684Z Version: 20241215.1.0
2024-12-16T16:40:46.7837631Z Included Software: https://github.com/actions/runner-images/blob/win22/20241215.1/images/windows/Windows2022-Readme.md
2024-12-16T16:40:46.7838991Z Image Release: https://github.com/actions/runner-images/releases/tag/win22%2F20241215.1
2024-12-16T16:40:46.7840242Z ##[endgroup]
2024-12-16T16:40:46.7841261Z ##[group]GITHUB_TOKEN Permissions
2024-12-16T16:40:46.7842891Z Contents: read
2024-12-16T16:40:46.7843409Z Metadata: read
2024-12-16T16:40:46.7843863Z Packages: read
2024-12-16T16:40:46.7844297Z ##[endgroup]
2024-12-16T16:40:46.7846921Z Secret source: Actions
2024-12-16T16:40:46.7847592Z ##[debug]Primary repository: ***/***
2024-12-16T16:40:46.7848217Z Prepare workflow directory
2024-12-16T16:40:46.7908402Z ##[debug]Creating pipeline directory: 'C:\a\***'
2024-12-16T16:40:46.7912530Z ##[debug]Creating workspace directory: 'C:\a\***\***'
2024-12-16T16:40:46.7914880Z ##[debug]Update context data
2024-12-16T16:40:46.7918147Z ##[debug]Evaluating job-level environment variables
2024-12-16T16:40:46.8115854Z ##[debug]Evaluating job container
2024-12-16T16:40:46.8118812Z ##[debug]Evaluating job service containers
2024-12-16T16:40:46.8121159Z ##[debug]Evaluating job defaults
2024-12-16T16:40:46.8144293Z Prepare all required actions
2024-12-16T16:40:46.8180459Z Getting action download info
2024-12-16T16:40:47.0495028Z Download action repository 'actions/checkout@v4' (SHA:11bd71901bbe5b1630ceea73d27597364c9af683)
2024-12-16T16:40:47.0578616Z ##[debug]Copied action archive 'C:\actionarchivecache\actions_checkout\11bd71901bbe5b1630ceea73d27597364c9af683.zip' to 'C:\a\_actions\_temp_77d28f9e-d1c4-4a3d-ae63-3c2b5add9e9b\0432622f-af9b-4cfe-9825-d03395055f4c.zip'
2024-12-16T16:40:47.1177177Z ##[debug]Unwrap 'actions-checkout-11bd719' to 'C:\a\_actions\actions\checkout\v4'
2024-12-16T16:40:47.1699031Z ##[debug]Archive 'C:\a\_actions\_temp_77d28f9e-d1c4-4a3d-ae63-3c2b5add9e9b\0432622f-af9b-4cfe-9825-d03395055f4c.zip' has been unzipped into 'C:\a\_actions\actions\checkout\v4'.
2024-12-16T16:40:47.1833111Z Download action repository 'actions/[email protected]' (SHA:1bd1e32a3bdc45362d1e726936510720a7c30a57)
2024-12-16T16:40:47.4163680Z ##[debug]Download 'https://api.github.com/repos/actions/cache/zipball/1bd1e32a3bdc45362d1e726936510720a7c30a57' to 'C:\a\_actions\_temp_e7bfaf1a-8e0a-4cb1-97aa-2be44230fd2a\127191c6-7540-4f59-a000-2d3d935a26a0.zip'
2024-12-16T16:40:47.5137154Z ##[debug]Unwrap 'actions-cache-1bd1e32' to 'C:\a\_actions\actions\cache\v4.2.0'
2024-12-16T16:40:47.5878838Z ##[debug]Archive 'C:\a\_actions\_temp_e7bfaf1a-8e0a-4cb1-97aa-2be44230fd2a\127191c6-7540-4f59-a000-2d3d935a26a0.zip' has been unzipped into 'C:\a\_actions\actions\cache\v4.2.0'.
2024-12-16T16:40:47.6051169Z Download action repository 'docker/[email protected]' (SHA:9780b0c442fbb1117ed29e0efdff1e18412f7567)
2024-12-16T16:40:47.8447404Z ##[debug]Download 'https://api.github.com/repos/docker/login-action/zipball/9780b0c442fbb1117ed29e0efdff1e18412f7567' to 'C:\a\_actions\_temp_c24a88d1-288b-4483-b6fa-645a767a2989\eefaa603-0ec6-4707-8610-a5137caf1478.zip'
2024-12-16T16:40:47.9064170Z ##[debug]Unwrap 'docker-login-action-9780b0c' to 'C:\a\_actions\docker\login-action\v3.3.0'
2024-12-16T16:40:47.9354894Z ##[debug]Archive 'C:\a\_actions\_temp_c24a88d1-288b-4483-b6fa-645a767a2989\eefaa603-0ec6-4707-8610-a5137caf1478.zip' has been unzipped into 'C:\a\_actions\docker\login-action\v3.3.0'.
2024-12-16T16:40:47.9405449Z Download action repository 'actions/setup-dotnet@v4' (SHA:3e891b0cb619bf60e2c25674b222b8940e2c1c25)
2024-12-16T16:40:48.1783669Z ##[debug]Download 'https://api.github.com/repos/actions/setup-dotnet/zipball/3e891b0cb619bf60e2c25674b222b8940e2c1c25' to 'C:\a\_actions\_temp_f956dbdf-b53a-4edf-bd1a-0d497761d038\eea64bcc-e283-456a-80e6-e1388ea871d8.zip'
2024-12-16T16:40:48.2615349Z ##[debug]Unwrap 'actions-setup-dotnet-3e891b0' to 'C:\a\_actions\actions\setup-dotnet\v4'
2024-12-16T16:40:48.3280162Z ##[debug]Archive 'C:\a\_actions\_temp_f956dbdf-b53a-4edf-bd1a-0d497761d038\eea64bcc-e283-456a-80e6-e1388ea871d8.zip' has been unzipped into 'C:\a\_actions\actions\setup-dotnet\v4'.
2024-12-16T16:40:48.3420374Z Download action repository 'aws-actions/configure-aws-credentials@v4' (SHA:e3dd6a429d7300a6a4c196c26e071d42e0343502)
2024-12-16T16:40:48.4711995Z ##[debug]Download 'https://api.github.com/repos/aws-actions/configure-aws-credentials/zipball/e3dd6a429d7300a6a4c196c26e071d42e0343502' to 'C:\a\_actions\_temp_f9bbd782-3859-4e96-8b45-1556e2cb4ef7\d9173f99-450d-4831-a378-67d23f11a006.zip'
2024-12-16T16:40:48.5112813Z ##[debug]Unwrap 'aws-actions-configure-aws-credentials-e3dd6a4' to 'C:\a\_actions\aws-actions\configure-aws-credentials\v4'
2024-12-16T16:40:48.5428440Z ##[debug]Archive 'C:\a\_actions\_temp_f9bbd782-3859-4e96-8b45-1556e2cb4ef7\d9173f99-450d-4831-a378-67d23f11a006.zip' has been unzipped into 'C:\a\_actions\aws-actions\configure-aws-credentials\v4'.
2024-12-16T16:40:48.5498172Z ##[debug]action.yml for action: 'C:\a\_actions\actions\checkout\v4\action.yml'.
2024-12-16T16:40:48.6148331Z ##[debug]action.yml for action: 'C:\a\_actions\actions\cache\v4.2.0\action.yml'.
2024-12-16T16:40:48.6221905Z ##[debug]action.yml for action: 'C:\a\_actions\docker\login-action\v3.3.0\action.yml'.
2024-12-16T16:40:48.6270378Z ##[debug]action.yml for action: 'C:\a\_actions\actions\setup-dotnet\v4\action.yml'.
2024-12-16T16:40:48.6333062Z ##[debug]action.yml for action: 'C:\a\_actions\aws-actions\configure-aws-credentials\v4\action.yml'.
2024-12-16T16:40:48.6499756Z ##[debug]Set step '__run' display name to: '***'
2024-12-16T16:40:48.6503450Z ##[debug]Set step '__actions_checkout' display name to: '***'
2024-12-16T16:40:48.6506256Z ##[debug]Set step '__actions_cache' display name to: '***'
2024-12-16T16:40:48.6509007Z ##[debug]Set step '__docker_login-action' display name to: 'Login (GitHub)'
2024-12-16T16:40:48.6511898Z ##[debug]Set step '__actions_setup-dotnet' display name to: '***'
2024-12-16T16:40:48.6515213Z ##[debug]Set step '__aws-actions_configure-aws-credentials' display name to: '***'
2024-12-16T16:40:48.6518688Z ##[debug]Set step '__run_2' display name to: '***'
2024-12-16T16:40:48.6521055Z ##[debug]Set step '__run_3' display name to: '***'
2024-12-16T16:40:48.6523641Z ##[debug]Set step '__run_4' display name to: '***'
2024-12-16T16:40:48.6526216Z ##[debug]Set step '__run_5' display name to: '***'
2024-12-16T16:40:48.6528696Z ##[debug]Set step '__run_6' display name to: '***'
2024-12-16T16:40:48.6531208Z ##[debug]Set step '__run_7' display name to: '***'
2024-12-16T16:40:48.6533829Z ##[debug]Set step '__run_8' display name to: '***'
2024-12-16T16:40:48.6536481Z ##[debug]Set step '__run_9' display name to: '***'
2024-12-16T16:40:48.6539081Z ##[debug]Set step '__run_10' display name to: '***'
2024-12-16T16:40:48.6542263Z Uses: ***/***/.github/workflows/***.yml@refs/heads/main (***)
2024-12-16T16:40:48.6545638Z ##[group] Inputs
2024-12-16T16:40:48.6546809Z   ***: ***
2024-12-16T16:40:48.6547653Z   ***: ***
2024-12-16T16:40:48.6549250Z   ***: ***
2024-12-16T16:40:48.6550056Z ##[endgroup]
2024-12-16T16:40:48.6551251Z Complete job name: *** / ***
2024-12-16T16:40:48.6645181Z ##[debug]Collect running processes for tracking orphan processes.
2024-12-16T16:40:48.6766419Z ##[debug]Finishing: Set up job
2024-12-16T16:41:24.3062620Z ##[debug]Evaluating condition for step: 'Login (GitHub)'
2024-12-16T16:41:24.3064675Z ##[debug]Evaluating: success()
2024-12-16T16:41:24.3065055Z ##[debug]Evaluating success:
2024-12-16T16:41:24.3065611Z ##[debug]=> true
2024-12-16T16:41:24.3065964Z ##[debug]Result: true
2024-12-16T16:41:24.3066493Z ##[debug]Starting: Login (GitHub)
2024-12-16T16:41:24.3150949Z ##[debug]Register post job cleanup for action: docker/[email protected]
2024-12-16T16:41:24.3175653Z ##[debug]Loading inputs
2024-12-16T16:41:24.3176832Z ##[debug]Evaluating: github.actor
2024-12-16T16:41:24.3177078Z ##[debug]Evaluating Index:
2024-12-16T16:41:24.3177298Z ##[debug]..Evaluating github:
2024-12-16T16:41:24.3177537Z ##[debug]..=> Object
2024-12-16T16:41:24.3177752Z ##[debug]..Evaluating String:
2024-12-16T16:41:24.3177955Z ##[debug]..=> 'actor'
2024-12-16T16:41:24.3178203Z ##[debug]=> 'austindrenski'
2024-12-16T16:41:24.3178433Z ##[debug]Result: 'austindrenski'
2024-12-16T16:41:24.3178933Z ##[debug]Evaluating: secrets.GITHUB_TOKEN
2024-12-16T16:41:24.3179171Z ##[debug]Evaluating Index:
2024-12-16T16:41:24.3179388Z ##[debug]..Evaluating secrets:
2024-12-16T16:41:24.3179608Z ##[debug]..=> Object
2024-12-16T16:41:24.3179795Z ##[debug]..Evaluating String:
2024-12-16T16:41:24.3179995Z ##[debug]..=> 'GITHUB_TOKEN'
2024-12-16T16:41:24.3180409Z ##[debug]=> '***'
2024-12-16T16:41:24.3180680Z ##[debug]Result: '***'
2024-12-16T16:41:24.3182010Z ##[debug]Loading env
2024-12-16T16:41:24.3185457Z ##[group]Run docker/[email protected]
2024-12-16T16:41:24.3185672Z with:
2024-12-16T16:41:24.3185827Z   username: austindrenski
2024-12-16T16:41:24.3186111Z   password: ***
2024-12-16T16:41:24.3186272Z   registry: ghcr.io
2024-12-16T16:41:24.3186434Z   ecr: auto
2024-12-16T16:41:24.3186578Z   logout: true
2024-12-16T16:41:24.3186720Z env:
2024-12-16T16:41:24.3186861Z   ***: ***
2024-12-16T16:41:24.3187039Z   ***: ***
2024-12-16T16:41:24.3187235Z   ***: 
2024-12-16T16:41:24.3187488Z   ***: ***
2024-12-16T16:41:24.3187904Z   ***: ***
2024-12-16T16:41:24.3188253Z   ***: ***
2024-12-16T16:41:24.3188466Z   ***: ***
2024-12-16T16:41:24.3188718Z   TARGET_BUILD_DATE_TIME: 2024-12-16T16:40:49.7896697Z
2024-12-16T16:41:24.3188963Z   TARGET_BRANCH: main
2024-12-16T16:41:24.3189137Z ##[endgroup]
2024-12-16T16:41:24.6663919Z Logging into ghcr.io...
2024-12-16T16:41:24.6675983Z ##[debug]Exec.getExecOutput: docker login --password-stdin --username austindrenski ghcr.io
2024-12-16T16:41:29.2672660Z ##[error]Error saving credentials: error storing credentials - err: exit status 1, out: `A specified logon session does not exist. It may already have been terminated.`
2024-12-16T16:41:29.2744000Z ##[debug]Node Action run completed with exit code 1
2024-12-16T16:41:29.2747840Z ##[debug]Save intra-action state isPost = true
2024-12-16T16:41:29.2748241Z ##[debug]Save intra-action state registry = ghcr.io
2024-12-16T16:41:29.2748596Z ##[debug]Save intra-action state logout = true
2024-12-16T16:41:29.2751044Z ##[debug]Finishing: Login (GitHub)

@mavimo
Copy link

mavimo commented Dec 16, 2024

@austindrenski this is related on how the secret is stored in the windows, see here, I suspect that during the upgrade we switch to a new version of docker that switch where we store secrets.

As workaround we add:

    - name: Tweak docker login
      shell: pwsh
      run: |
        New-Item -Path $HOME/.docker -ItemType Directory -Force
        Set-Content -Path $HOME/.docker/config.json -Value '{"auths": {"XXXXXXXXXX": {}}}'

where XXXXXXXXXX is the docker registry relevant for you (e.g. quay.io, us-docker.pkg.dev, ...).

@austindrenski
Copy link

Thanks @mavimo, that does indeed get us unblocked in the meantime!


Note

In the spirit of contributing copy/pasta for future readers, here's what we added just before the first login action step of each affected workflow:

- name: Temporary workaround for actions/runner-images#11172
  run: |
    mkdir --parents ~/.docker
    echo '{"auths":{"ghcr.io":{}}}' > ~/.docker/config.json
  shell: bash

@maxim-lobanov
Copy link
Contributor

Hello everyone,

Thank you for reporting the problem. We have identified root cause and working on fix.
The fix should be deployed within 1-2 days. Also, we are discussing internally a hotfix which can be deployed faster.

@maxim-lobanov
Copy link
Contributor

Fix is deployed and I confirmed that issue is resolved. Could you please double check that issue is resolved for you too?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
awaiting-deployment Code complete; awaiting deployment and/or deployment in progress bug report OS: Windows
Projects
None yet
Development

No branches or pull requests

6 participants