Endpoint auth. parameter that is not confidential getting masked in agent logs

[GitHubSriramB]

2.122.0

Windows

VSTS

Attaching details about the custom endpoint type & logs that has this issue.

Endpoint details logged in agent:

{
  "data": {},
  "id": "cfa7e53f-5778-423c-af65-507a80224c79",
  "name": "cfa7e53f-5778-423c-af65-507a80224c79",
  "type": "habitatoriginendpoint",
  "url": "https://bldr.habitat.sh",
  "authorization": {
    "parameters": {
      "username": "********",
      "revision": "********",
      "publickey": "********",
      "password": "********",
      "githubauthtoken": "********",
      "useSudo": "********"
    },
    "scheme": "UsernamePassword"
  },
  "isReady": false
}

Endpoint contribution type:

{
  "id": "habitat-origin",
  "description": "Habitat Origin",
  "type": "ms.vss-endpoint.service-endpoint-type",
  "targets": ["ms.vss-endpoint.endpoint-types"],
  "properties": {
    "name": "habitatoriginendpoint",
    "displayName": "Habitat Origin",
    "url": {
      "displayName": "Habitat Depot URL",
      "helpText": "URL to the Habitat depot that will be used to deploy to"
    },
    "inputDescriptors": [],
    "authenticationSchemes": [
      {
        "type": "ms.vss-endpoint.endpoint-auth-scheme-basic",
        "inputDescriptors": [
          {
            "id": "username",
            "name": "Origin Name",
            "description": "Name of the Habitat origin",
            "inputMode": "textbox",
            "isConfidential": false,
            "validation": {
              "isRequired": true,
              "dataType": "string"
            }
          },
          {
            "id": "revision",
            "name": "Revision",
            "description": "Revision of the origin to use",
            "inputMode": "textbox",
            "isConfidential": false,
            "validation": {
              "isRequired": true,
              "dataType": "string"
            }
          },
          {
            "id": "publickey",
            "name": "Public Key",
            "description": "Public item of the origin key pair",
            "inputMode": "textarea",
            "isConfidential": false,
            "validation": {
              "isRequired": true,
              "dataType": "string"
            }
          },
          {
            "id": "password",
            "name": "Signing Key",
            "description": "Signing item of the origin key pair",
            "inputMode": "textarea",
            "isConfidential": true,
            "validation": {
              "isRequired": true,
              "dataType": "string"
            }
          },
          {
            "id": "githubauthtoken",
            "name": "GitHub Auth Token",
            "description": "Authentication token for GitHub for publishing Habitat packages.",
            "inputMode": "textbox",
            "isConfidential": "true",
            "validation": {
              "isRequired": true,
              "dataType": "string"
            }
          },
          {
            "id": "useSudo",
            "name": "Use Sudo",
            "description": "Use sudo on habitat commands",
            "inputMode": "combo",
            "isConfidential": false,
            "validation": {
              "isRequired": false,
              "dataType": "string"
            },
            "values": {
              "inputId": "useSudoValues",
              "defaultValue": "1",
              "possibleValues": [
                {
                  "value": "1",
                  "displayValue": "True"
                },
                {
                  "value": "0",
                  "displayValue": "False"
                }
              ]
            }
          }
        ]
      }
    ],
    "helpMarkDown": "Please provide the requested information so that Habitat packages can be built and deployed. The GitHub authentication token requires `user:email` and `read:org` scopes"
  }
}

[bryanmacfarlane]

Are you sure this is an agent issue (and not lib/service)?

[GitHubSriramB]

@bryanmacfarlane - I was referring to this piece of code in agent that needs a fix:
https://github.com/Microsoft/vsts-agent/blob/327cde0be616c9809168b160f9431702b7b46e1c/src/Agent.Worker/Worker.cs#L144

[GitHubSriramB]

Fix for this issue is blocked due to a service-side bug in Build. Waiting for the service-side bug to be fixed before we can take up this fix.

@ericsciple - pl. assign this back to us once the service-side bug is fixed.

[chperich]

This feedback ticket has been raised recently by a user. Character 1 gets masked in the user's logs because of masking all endpoint authorization parameters.

[peterox]

Is there any update to this issue? I am also having the problem where all '1' characters are being masked.

[chperich]

+ Lova/Omesh, who will be able to address this better. From: Peter Oxenham <[email protected]> Sent: Wednesday, December 12, 2018 9:50 AM To: Microsoft/azure-pipelines-agent <[email protected]> Cc: Chandana Pericharla <[email protected]>; Comment <[email protected]> Subject: Re: [Microsoft/azure-pipelines-agent] Endpoint auth. parameter that is not confidential getting masked in agent logs (#1207) Is there any update to this issue? I am also having the problem where all '1' characters are being masked. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoft%2Fazure-pipelines-agent%2Fissues%2F1207%23issuecomment-446456779&data=02%7C01%7Cchperich%40microsoft.com%7C0ed0afe3b6d240e565a508d65fe916c2%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636801852050381035&sdata=oxESjX0NhVoQGh%2BdLQ%2B0IQ0OABC1NNMvWpL%2F0TKRido%3D&reserved=0>, or mute the thread<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAbtqD33iOswRcL7CPtn-j4v_p5bsYOZ7ks5u4IPzgaJpZM4PXFhR&data=02%7C01%7Cchperich%40microsoft.com%7C0ed0afe3b6d240e565a508d65fe916c2%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636801852050381035&sdata=J2OiyI46epiJY96F4U1X1CoMgwXKhjjb3P1nFSS2GOA%3D&reserved=0>.

[omeshp]

@ericsciple should be able to provide update on build side issue.

[ericsciple]

@peterox I believe this was fixed. Are you using the service "Azure Devops" and the latest agent? Or are you using on-premises TFS? Note you can right click on the pool to update all agents in a pool to the latest version.

[peterox]

@ericsciple I've just updated all the agents and I'm still getting all '1''s masked in the logs.

I disabled a Docker task that had an Azure subscription configured and ran the build and the issue was not there. Re-enabling the task and running the job again caused the issue to re-appear.

[ericsciple]

@TingluoHuang do you remember context on this? it looks like this code is still in master. I thought this was removed months ago?

[peterox]

Is there a work around available to prevent masking?

[peterox]

Bump

[willgarcia]

Just confirming that the latest Github released version (2.144.2) does not fix this issue after redeployment of my agents.

The HTTP link in the Web UI is also pointing to version 2.144.0 (instead of 2.144.2).

Current version of worker.cs on master is from Aug 17, 2018 (ab40f0b1..).

@ericsciple Would you be able to (un)confirm if the bug is still on master or a fixed on another branch?

[deadlydog]

We are using hosted Azure DevOps in the cloud with on-prem agents running version vsts-agent-win-x64-2.144.0 and we are also seeing this problem still.

[peterox]

Are we able to get an update on this? It seems there is a fix but it just hasn't been released?

This problem effectively makes the logs in our pipelines useless.

[willgarcia]

As spotted by @peterox, removing our Azure DevOps Docker tasks from pipelines seems to solve the problem.

Specific tasks types that require access to secrets may interfere and cause this issue.

[KonstantinTyukalov]

Hey everyone!
We are introducing new agent knob: AGENT_IN_SECRET_LENGTH which will let you not to mask short secrets.
You can specify values from 0 (all secrets will be masked as usual) to 4 (secrets shorten than 4 symbols will be ignored)
Also, negative values will mean masking all the secrets as usual.

example:

  variables:
    AGENT_MIN_SECRET_LENGTH: 1

also it's available as environment variable.

This feature will be available in the next agent release, we will notify you once it will be rolled out.
Thanks!

UPD: The new knob name is AZP_IGNORE_SECRETS_SHORTER_THAN

[StingyJack]

@KonstantinTyukalov is it going to stop the changing of dates and times on the logs or the header information for tasks?
There should be nothing masking these areas of the logs as secret values. This is a large part of the problem because that is the information that can be used to guess the secret values.

The request was for you to not mask trivial (single character) or values that are well known to not be secret, like "Dev" or "32" or "Q".

A length setting is not useful if the agent is still possibly going to mask easily discernable values like those. Is this an opt-in setting?

I think you guys missed the point still. It's not about not masking small values, the problem is you're masking values that are not secrets that help me reliably guess the secrets.

Also the name of the variable AGENT_IN_SECRET_LENGTH does not make sense. I would not be able to use that in a sentence explaining what it does.

[DenisRumyantsev]

@StingyJack I have tested that timestamps remain and are not affected by secret masking:

Also, please notice that the name of the variable is AGENT_MIN_SECRET_LENGTH , not AGENT_IN_SECRET_LENGTH

[StingyJack]

I have tested that timestamps remain and are not affected by secret masking

@DenisRumyantsev - Yes and I notice the log is now totally useless as well.

Also, please notice that the name of the variable is AGENT_MIN_SECRET_LENGTH , not AGENT_IN_SECRET_LENGTH

That makes more sense but its not what @KonstantinTyukalov said it was called.

We are introducing new agent knob: AGENT_IN_SECRET_LENGTH which will let you not to mask short secrets.

That name still doesnt refer to the masking which its affecting. Which is not as important as that you are still missing the point. We dont want you to avoid masking secrets. This solution makes the problem worse as it allows me to expose secrets directly instead of just inferring them indirectly.

The current design may be simple but it is far too aggressive, We want you to avoid masking data that is not a secret.

01 - 2022-03-12T23:40:40.2210221Z ##[section]Starting: Use Node 1***.18.***
02 - 2022-03-12T23:40:40.2244542Z ==============================================================================
03 - 2022-03-12T23:40:40.2245097Z Task         : Node.js tool installer
04 - 2022-03-12T23:40:40.2245714Z Description  : Finds or downloads and caches the specified version spec of Node.js and adds it to the PATH
05 - 2022-03-12T23:40:40.2246490Z Version      : 0.***00.0
06 - 2022-03-12T23:40:40.2246928Z Author       : Microsoft Corporation
07 - 2022-03-12T23:40:40.2247457Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/tool/node-js
08 - 2022-03-12T23:40:40.2248059Z ==============================================================================
09 - 2022-03-12T23:40:40.3932609Z Downloading: https://nodejs.org/dist/v1***.18.***/node-v1***.18.***-linux-x64.tar.gz
10 - 2022-03-12T23:40:40.6562110Z Extracting archive
11 - 2022-03-12T23:40:40.6587983Z [command]/usr/bin/tar xC /home/vsts/work/_temp/14775***7***-0fb9-4***8***-bf57-***ae8199eab81 -f /home/vsts/work/_temp/090***8e***c-174b-4ac***-8a76-f95b8***8c88a5
12 - 2022-03-12T23:40:41.2737568Z Caching tool: node 1***.18.*** x64
13 - 2022-03-12T23:40:41.8463078Z Prepending PATH environment variable with directory: /opt/hostedtoolcache/node/1***.18.***/x64/bin
14 - 2022-03-12T23:40:41.9082799Z ##[section]Finishing: Use Node 1***.18.***

Consider the following...
If logging the Task header info (lines 2-8), dont apply masking because secrets cannot be exposed here by anyone authoring a pipeline
If logging an agent-emitted section starting (line 1) or finishing (line 14), dont apply masking because secrets cannot be exposed here by anyone authoring a pipeline.
If the task configuration does not reference any secrets, dont apply masking because no secrets are being used in the task.
If the task configuration does reference secrets, mask only those secret values that are referenced, and only in the places where the reference would be emitted
etc..

[StingyJack]

@DenisRumyantsev @KonstantinTyukalov @anatolybolshakov @alexander-smolyakov - I pointed out that your proposed solution creates an even larger problem than was being reported, and you have nothing to say in the last month?

I really hope that you are not still considering this AGENT_SECRET_MIN_LENGTH as viable, because nobody asked for you for a knob that let us expose any secrets. We asked you to stop making the secret values easy to guess.

[at-besa]

This issue is open for years and no one has yet figured out what's going on!? 🙄

In our pipelines, we are using Variable Groups connected to a Key Vault, right after the Download Secret task, all 1s are replaced with ***.

Have checked all the secrets in the Key Vault. None of them is 1. Of course, it could be part of a secret like something_1_something_else, but not by itself as a whole._

---

Hey everyone! We are introducing new agent knob: AGENT_IN_SECRET_LENGTH which will let you not to mask short secrets. You can specify values from 0 (all secrets will be masked as usual) to 4 (secrets shorten than 4 symbols will be ignored) Also, negative values will mean masking all the secrets as usual._

---

The knob is not working for the bug where that '1' is masked, we also checked our secrets and there is none of them having a 1. Also interesting fact is that it appears to happen only on the first stage, if the pipeline does have more stages the 1's are not masked except the first stage. The pipelines use the same agent for these stages, and the agent version is 2.217.2, which is the newest as by the time writing this.

So result is neither is the bug with suddenly masking 1 solved nor is there any workaround or solution to not have guessable logs where secrets are masked out of text which shouldn't be masked.

@KonstantinTyukalov @anatolybolshakov

[at-besa]

This issue is open for years and no one has yet figured out what's going on!? 🙄
In our pipelines, we are using Variable Groups connected to a Key Vault, right after the Download Secret task, all 1s are replaced with ***.
Have checked all the secrets in the Key Vault. None of them is 1. Of course, it could be part of a secret like something_1_something_else, but not by itself as a whole._

Hey everyone! We are introducing new agent knob: AGENT_IN_SECRET_LENGTH which will let you not to mask short secrets. You can specify values from 0 (all secrets will be masked as usual) to 4 (secrets shorten than 4 symbols will be ignored) Also, negative values will mean masking all the secrets as usual._

The knob is not working for the bug where that '1' is masked, we also checked our secrets and there is none of them having a 1. Also interesting fact is that it appears to happen only on the first stage, if the pipeline does have more stages the 1's are not masked except the first stage. The pipelines use the same agent for these stages, and the agent version is 2.217.2, which is the newest as by the time writing this.

So result is neither is the bug with suddenly masking 1 solved nor is there any workaround or solution to not have guessable logs where secrets are masked out of text which shouldn't be masked.

@KonstantinTyukalov @anatolybolshakov

This also happens with the new Azure Agent with Version 3.217.1 as we just tried the pre-release version.

[KonstantinTyukalov]

@at-besa Do you have this or other extension in your pipeline? #1207 (comment)

Also, have you tried AZP_IGNORE_SECRETS_SHORTER_THAN knob name? Note that AGENT_MIN_SECRET_LENGTH knob name is not valid for now.

Also interesting fact is that it appears to happen only on the first stage, if the pipeline does have more stages the 1's are not masked except the first stage.

What tasks do you have in this first stage?

[StingyJack]

Also, have you tried AZP_IGNORE_SECRETS_SHORTER_THAN knob name? Note that AGENT_MIN_SECRET_LENGTH knob name is not valid for now.

@KonstantinTyukalov - this knob does not fix the problem that was reported. It actually makes the situation worse, thus invalidating it's intended purpose. Several users have told you this now. Please revert the knob and come up with a proper fix.

[at-besa]

@at-besa Do you have this or other extension in your pipeline? #1207 (comment)

Also, have you tried AZP_IGNORE_SECRETS_SHORTER_THAN knob name? Note that AGENT_MIN_SECRET_LENGTH knob name is not valid for now.

Also interesting fact is that it appears to happen only on the first stage, if the pipeline does have more stages the 1's are not masked except the first stage.

What tasks do you have in this first stage?

Yes i have tried the knob, the 1's are not masked anymore when i turn the knob value of AZP_IGNORE_SECRETS_SHORTER_THAN to 1, although this does not make any sense, as mentioned there is no secret which does even have this value. I agree with @StingyJack that this is just an workaround and not the solution to the actual problem.

regarding the tasks used in the first stage:
- checkout: self
- task: Bash@3
- task: Docker@2

The whole pipeline is running on an azure hosted vm with rocky as operating system.

as there is no secret stuff in this section i can share the yaml of this whole stage:

  - stage: Build
    jobs:
      - job: Build
        timeoutInMinutes: 10
        cancelTimeoutInMinutes: 2
        steps:
          - checkout: self
            fetchDepth: 2
            clean: true
            submodules: recursive
             
          ############ convert repo name to lowercase for openshift  ############
          - task: Bash@3
            displayName: 'Convert Repo to lowercase'
            inputs:
              targetType: 'inline'
              script: |
                LowerRepoName=$(echo $(Build.Repository.Name) | tr  [:upper:] [:lower:])
                echo "##vso[task.setvariable variable=RepoName]$LowerRepoName"
          
          
          ############ build image, tag and push it to registry  ############
          - task: Docker@2
            displayName: 'docker build and push'
            inputs:
              containerRegistry: 'openshift-registry'
              repository: '$(imageName)'
              command: 'buildAndPush'
              Dockerfile: '**/Dockerfile'
              tags: |
                $(Build.BuildNumber)
                latest

[at-besa]

any news on that ?

[github-actions]

This issue has had no activity in 180 days. Please comment if it is not actually stale

[at-besa]

180 days and still no comment of devs

[github-actions]

This issue has had no activity in 180 days. Please comment if it is not actually stale

[StingyJack]

Turn off this bot. It just makes the maintainers come across as "rude" instead of just "busy"

[github-actions]

This issue has had no activity in 180 days. Please comment if it is not actually stale

[StarWars999123]

Turn off this bot. It just makes the maintainers come across as "rude" instead of just "busy"

would consider them extremly ignorant now... This issue annoys me almost every day.

[roospa]

We have been monitoring this issue for a quite some while now...amazing....still today the issue is not fixed and logs are becoming fairly unusable as much of non secret information is being masked.
2024-11-06T13:54:19.5517066Z Current agent version: '3.242.1'