CodeQL adds redundant slash to upload sarif file endpoint

[filipss379]

I use Azure DevOps pipeline to perform CodeQL code scanning. It goes well until I try to upload sarif file to github. I use this command to do upload: echo $(Github.TOKEN) | $(codeql) github upload-results --repository=$(Owner)/$(Repo) --ref=$(Build.SourceBranch) --commit=$(Build.SourceVersion) --sarif=$(sarifOutput) --github-auth-stdin --github-url=https://github.com. In result CodeQL tries to upload sarif file using endpoint https://api.github.com//repos/owner_name/repo_name/code-scanning/sarifs. There is redundant slash in the url so I get 404 response. Do you have idea what is wrong and how to fix?

[jketema]

Hi @filipss379,

Which version of CodeQL are you using?

[filipss379]

2.15.3

[smowton]

It is true that a redundant / is present in the URL, but this is only cosmetic. The 404 error most likely results from github.com not having a ref corresponding to your --ref parameter, or commit corresponding to your --commit, or repository $(Owner)/$(Repo). I recommend you should print those values as incorporated into the command and check whether the relevant refs, commits and repository exist as printed on github.com.

[filipss379]

Thank you for you reply. I checked all that parameters and it is correct - there is corresponding branch and commit in the repository. I did one additional test - I tried call the endpoint https://api.github.com/repos/owner/repo/code-scanning/sarifs through postman (just left "sarif" property empty in payload) and I got 400 response but when I added redundant slash in url and sent the request again I got 404 as I'm getting in pipeline so I think redundant slash is an issue here.

[smowton]

I think that's true, but I think the URL is getting cleaned up in between the URL object used to print the 404 error and being used to make the request -- my basis for this is having made a successful upload using codeql 2.15.3 using the same options as you, and then getting a 404 if I alter any of the ref, commit or repository parameters.

Can you confirm the commandline you're using with all the variables substituted, editing minimally to remove anything confidential but keeping as much of the syntax as possible?

[smowton]

Could you also paste the exact error you get please, again redacting as minimally as possible?

[filipss379]

So here is command that I use echo $(Github.TOKEN) | codeql github upload-results --repository=Volue/PrivateFacilities.Discharge --ref=refs/heads/master --commit=5d34c280cb950166481c43dc5f71aab347baf7d0 --sarif=codeqlAnalisys/web.sarif --github-auth-stdin --github-url=https://github.com.
I get following error:
A fatal error occurred: Error uploading SARIF to 'https://api.github.com//repos/Volue/PrivateFacilities.Discharge/code-scanning/sarifs' from 'D:\a\1\s\codeqlAnalisys\web.sarif'. REASON: HTTP/1.1 404 Not Found:::{"message":"Not Found","documentation_url":"https://docs.github.com/rest/code-scanning/code-scanning#upload-an-analysis-as-sarif-data"}

[smowton]

One reason for getting a 404 could be that token not having the needed rights to use that API endpoint. Is that an automatic token (https://docs.github.com/en/actions/security-guides/automatic-token-authentication), and if so do you know whether it has the needed rights? Could you try uploading sarif using a personal access token that has the security_events scope (classic) or the Code scanning alerts permission (fine-grained token)?

[filipss379]

The token I use is fine-grained token and it has write and read access for code scanning alerts.

[smowton]

I note the API docs at https://docs.github.com/en/rest/code-scanning/code-scanning?apiVersion=2022-11-28#upload-an-analysis-as-sarif-data say You must use an access token with the security_events scope to use this endpoint for private repositories. I would expect your fine-grained token to work too, but are you able to experimentally try a classic token to see if it makes a difference? I would suggest initially making the token with broad permissions that enable you to verify it can see the private repository in question and has write access (e.g., verifying it can report a check run against a commit), then incrementally narrow its permissions to the minimum that should be necessary.

[filipss379]

I generated classic token with security_events scope and tried with this one. Same result - 404. But then I took sarif file generated in pipeline and used postman to upload it calling https://api.github.com/repos/Volue/PrivateFacilities.Discharge/code-scanning/sarifs as shown here https://docs.github.com/en/rest/code-scanning/code-scanning?apiVersion=2022-11-28#upload-an-analysis-as-sarif-data with the same parameters as I pasted above and I got 202 accepted (tried with fine-grained token and classic token and both worked). I see now in my repo the code scanning result. So scope and type of a token is not an issue here.

[smowton]

For reproduction purposes, could you paste the exact command that got you a 202?

[smowton]

FYI current status here: it appears the leading / really is making a difference, which I have to admit I had thought implausible considering this command at least sometimes works, and adding a leading / seems like it should be all-or-nothing bug. We're going to make a fix our end and provide an (unsuported/experimental) nightly build of the CodeQL CLI to try out and verify the fix works for you in advance of the regular release cycle.

[filipss379]

Great to here you found the issue. For now to make upload working I'm adding token to environment variables like this ##vso[task.setvariable variable=GITHUB_TOKEN]$(Github.TOKEN) and then I run command $(codeql) github upload-results --repository=$(Owner)/$(Repo) --ref=$(Build.SourceBranch) --commit=$(Build.SourceVersion) --sarif=$(sarifOutput) --github-url=https://github.com skipping --github-auth-stdin option.

[smowton]

Could you test whether CodeQL nightly https://github.com/dsp-testing/codeql-cli-nightlies/releases/download/codeql-bundle-20231211/codeql-bundle.tar.gz fixes your problem?

[AdamSmith-BT]

I have similar symptoms, but not sure if it's the same problem. How did you confirm the URL that the CodeQL CLI was using?

In my environment I have tested the above nightly build but that didn't resolve the issue.

[smowton]

@AdamSmith-BT can you share the log of the failure you're seeing?