Issues using published codeql pack

[jacob-ronstadt]

Description of the issue
I published a custom CodeQL pack and am trying to test it on a new machine but I am running into some unexpected behavior. I have reproduced the issue with codeql cli version 2.15.4 and 2.16.0

in the qlpack.yml file I have the following dependencies

dependencies:
    codeql/cpp-queries: 0.9.0
    codeql/cpp-all: 0.12.1

(I also tried using "*" as the version instead of specifying a version and get the same result)

I run codeql pack install to install these to my machine before publishing.

I then run codeql pack publish.

On the new machine I run codeql pack download <name for my pack> and it successfully downloads the pack.

I then try to use the new pack to analyze a database but get an error stating codeql/[email protected] cannot be found.

A fatal error occurred: The QL pack 'codeql/[email protected]' which is referenced from <my packs default query suite> cannot be found.

After this I try running codeql pack install on the install directory of my codeql pack. This generates several warnings about packs being found via "--additional-packs" and nothing gets installed.

WARNING: Pack 'codeql/cpp-all' was found via '--additional-packs'. If the generated 'codeql-pack.lock.yml' file is committed to source control, other users will also have to use '--additional-packs' when compiling or running queries.  (<pack install location>\.codeql\libraries\codeql\cpp-all\0.12.3\qlpack.yml:1,1-1)
WARNING: Pack 'codeql/dataflow' was found via '--additional-packs'. If the generated 'codeql-pack.lock.yml' file is committed to source control, other users will also have to use '--additional-packs' when compiling or running queries.  (<pack install location>\.codeql\libraries\codeql\dataflow\0.1.6\qlpack.yml:1,1-1)
WARNING: Pack 'codeql/rangeanalysis' was found via '--additional-packs'. If the generated 'codeql-pack.lock.yml' file is committed to source control, other users will also have to use '--additional-packs' when compiling or running queries.  (<pack install location>\.codeql\libraries\codeql\rangeanalysis\0.0.5\qlpack.yml:1,1-1)
WARNING: Pack 'codeql/ssa' was found via '--additional-packs'. If the generated 'codeql-pack.lock.yml' file is committed to source control, other users will also have to use '--additional-packs' when compiling or running queries.  (<pack install location>.codeql\libraries\codeql\ssa\0.2.6\qlpack.yml:1,1-1)
WARNING: Pack 'codeql/tutorial' was found via '--additional-packs'. If the generated 'codeql-pack.lock.yml' file is committed to source control, other users will also have to use '--additional-packs' when compiling or running queries.  (<pack install location>.codeql\libraries\codeql\tutorial\0.2.6\qlpack.yml:1,1-1)
WARNING: Pack 'codeql/typetracking' was found via '--additional-packs'. If the generated 'codeql-pack.lock.yml' file is committed to source control, other users will also have to use '--additional-packs' when compiling or running queries.  (<pack install location>.codeql\libraries\codeql\typetracking\0.2.6\qlpack.yml:1,1-1)
WARNING: Pack 'codeql/util' was found via '--additional-packs'. If the generated 'codeql-pack.lock.yml' file is committed to source control, other users will also have to use '--additional-packs' when compiling or running queries.  (<pack install location>\.codeql\libraries\codeql\util\0.2.6\qlpack.yml:1,1-1)
WARNING: To avoid these warnings in the future, use the '--no-strict-mode' option. (<pack install location>\qlpack.yml:1,1-1)
Dependencies resolved. Installing packages...
Install location: C:\Users\Administrator\.codeql\packages
Nothing to install.
Package install location: C:\Users\Administrator\.codeql\packages
Nothing downloaded.

Trying to analyze a database again results in the same error that codeql/[email protected] cannot be found.

However, if again run codeql pack install on the install directory of my codeql pack, it runs without warnings, the dependencies are downloaded, and I can run analysis.

If I try using the "--no-strict-mode" flag when running the install command, it will again not install anything the first time (this time without warnings) and I get errors trying to analyze, then if I rung the command a second time it will download the dependencies. (same behavior as without --no-strict-mode but this time without showing warnings)

My main questions are:

1. Why I get these warnings when trying to install dependencies
2. Why I need to run the command twice to get the expected behavior
3. If this behavior actually is expected, is something specific I need to do to avoid it

[aeisenberg]

It looks like you are trying to run a pack that you have downloaded by passing the full path to the pack. What command are you using precisely?

Please include all of the commands you are running (redacted paths are ok). I think there is something incorrect about some of the options you are using.

[aeisenberg]

Also, unrelated, I see you have codeql/cpp-queries as a dependency. Normally, this is not recommended unless your pack has tests. Is there a specific reason for this?

[jacob-ronstadt]

It looks like you are trying to run a pack that you have downloaded by passing the full path to the pack. What command are you using precisely?

Please include all of the commands you are running (redacted paths are ok). I think there is something incorrect about some of the options you are using.

To download:
codeql pack download microsoft/windows-drivers

To install dependencies:
codeql pack install C:\Users\Administrator.codeql\packages\microsoft\windows-drivers\1.0.2\

I also found that running "codeql pack resolve-dependencies C:\Users\Administrator.codeql\packages\microsoft\windows-drivers\1.0.2" Generates the same warnings about "--additional-packs" and then "codeql pack install C:\Users\Administrator.codeql\packages\microsoft\windows-drivers\1.0.2" doesn't have warnings and installs the dependencies. But either way I can't just download and run install once. I either have to download, resolve dependencies, install, or download, install, install.

[jacob-ronstadt]

Also, unrelated, I see you have codeql/cpp-queries as a dependency. Normally, this is not recommended unless your pack has tests. Is there a specific reason for this?

The query suites we require to be run contain queries from codeql/cpp-queries so it was added as a dependency. If there is a better way to do that, let me know.

[aeisenberg]

You should not need to run codeql pack install on a pack you downloaded from the registry. The reason is that published packs all have their own dependencies bundled with them.

Instead you can just run codeql database analyze ... microsoft\windows-drivers. The CLI knows how to find packs that were downloaded into the .codeql\packages directory.

[jacob-ronstadt]

If I try running analyze without first running install, I get an error that codeql/cpp-queries cannot be found

C:\Users\Administrator>codeql database analyze C:\Users\Administrator\Desktop\faildriver_test2 microsoft/windows-drivers --format=sarifv2.1.0 --output=out.sarif
Running queries.
Stingpool size measured as 5800330
A fatal error occurred: The QL pack 'codeql/[email protected]' which is referenced from C:\Users\Administrator\.codeql\packages\microsoft\windows-drivers\1.0.2\suites\windows_driver_mustfix.qls cannot be found.

[aeisenberg]

Right. I was just answering your second question, saying that you're going to have a problem when you reference queries from other packs in a query suite.

Let me think of a good solution for you.

[jacob-ronstadt]

Also, if I go into the directory where the package was installed, I see that there is a folder for cpp-queries which contains the .ql queries. So it looks like they exist in the qlpack but when running analyze it doesn't find them

C:\Users\Administrator\.codeql\packages\microsoft\windows-drivers\1.0.2\.codeql\libraries\codeql\cpp-queries\0.9.0

[aeisenberg]

Create a qls file that lives outside of the the package registry. This qls file can reference queries and suites from any published query pack. Instead of downloading codeql/cpp-queries and microsoft\windows-drivers explicitly, you can do something like this:

codeql database analyze --download ... path/to/local/suite.qls

This will download all of the suite's dependencies before trying to run anything.

You were hitting two limitations before:

1. query packs do not include queries from other packs.
2. query suites in query packs cannot depend on other query packs.

[NateD-MSFT]

Jacob's coworker here - I just wanted to give some more context for why our suite is including stuff from cpp-queries. We work on the Windows Hardware Certification Program at Microsoft and as part of that we enforce that developers run CodeQL on their drivers ahead of submission. We have a number of custom queries (see https://github.com/microsoft/Windows-Driver-Developer-Supplemental-Tools) that we require people to run, but we also require developers to run some of the standard CodeQL C++ queries that can find bugs and security issues.

Ideally, we'd like to be able to have users download our single QL pack, and then have access to the query suites that cover what we require and recommend across both our custom queries and the standard C++ queries via the package automatically downloading+installing dependencies. The method you suggest is doable, but in a perfect world we'd love to leave it all up to the CodeQL CLI.

[aeisenberg]

Thanks for the context. We're having a discussion internally to see if we can better support your use case.

[NateD-MSFT]

Much appreciated! Please keep us in the loop.

[jacob-ronstadt]

Hi @aeisenberg, I've been doing some more testing and I found that installing the codeql-cli bundle instead of the standalone codeql-cli results in the behavior I originally expected.

Using the bundle, and with codeql/cpp-queries and codeql/cpp-all dependencies in the qlpack, I can run the command to download codeql pack microsoft/[email protected] and then analyze a database with that pack without issues.

My questions are:

1. Besides including precompiled versions of the codeql queries, is there anything different between codeql-cli from github.com/github/codeql-action and github.com/github/codeql-cli?
2. Is there a reason using the bundle like this would cause issues?

[aeisenberg]

When you are using the bundle, the queries you are referencing are resolved from the bundle, not from the qlpack that you produce. I'm actually a little surprised this is working since we should be marking errors if a query suite inside a query pack tries to reference queries in another pack.

Other than including compiled queries, the bundle and the cli are the same.

If you are ok using the bundle, then I would recommend that you avoid including the queries directly in your query pack.

[aeisenberg]

Is there a reason using the bundle like this would cause issues?

If you are able to resolve queries in the bundle from your query suite, then there should be no problem using it.

[jacob-ronstadt]

I did end up getting errors when using the bundle and referencing codeql/cpp-queries in my qlpack. It looks like if I'm using the bundle, and have my published qlpack downloaded to my machine, if the source of my qlpack also exists on my machine, codeql will use that instead of the qlpack and I don't have issues with referencing other qlpacks.

If I remove the sources for my qlpack, codeql uses the downloaded one and will give me the error that a compiled qlpack cannot reference another pack from one of its query suites.

Will codeql always use a local version/source code of a qlpack over the downloaded/compiled version? Is there a way to specify which you want to use?

[aeisenberg]

Will codeql always use a local version/source code of a qlpack over the downloaded/compiled version?

That's correct. A source pack always overrides a compiled pack.

Is there a way to specify which you want to use?

No, except by specifying different --additional-packs or by not using a bundle.