Skip to content

Latest commit

 

History

History
340 lines (232 loc) · 15.4 KB

GUIDE.md

File metadata and controls

340 lines (232 loc) · 15.4 KB
Raw

Integration Guide

Explanation

The meta-fossa layer for Yocto integrates FOSSA CLI as a build step inside the Yocto build system.

The meta-fossa layer does not actually alter the build, but does need to be run as part of the Yocto build process so that FOSSA can query Yocto for dependency and project information.

Integration Steps

  1. Clone Yocto (if not already cloned) and meta-fossa:
git clone git://git.yoctoproject.org/poky.git -b dunfell
git clone https://github.com/fossas/meta-fossa.git
  1. Add the layer to Bitbake:
cd poky
source oe-init-build-env
bitbake-layers add-layer ../../meta-fossa/
  1. Add meta-fossa to build/conf/local.conf by appending the following lines:
BBLAYERS += "<PATH-TO-META-FOSSA>"
INHERIT += "fossa"
FOSSA_API_KEY = "<VALID-FOSSA-API-KEY>"
  1. Run the build:
bitbake core-image-minimal

Analysis

meta-fossa currently supports the following strategies:

Strategy License Description Homepage Authors Excludes unused packages Vulnerability
custom-deps ✅ ✅ ✅ ❌ ✅ ❌
vendored-deps ✅ ❌ ❌ ❌ ✅ ❌

More strategies are currently in active development.

custom-deps strategy

In this strategy, the fossa class does the following:

  1. Parses each recipe and persists JSON-ified recipe in tmp/fossa_metadata/recipes/.
  2. Identifies runtime packages via oe.rootfs.image_list_installed_packages.
  3. Correlates runtime packages to recipe and package's metadata using a similar approach as scripts/oe-pkgdata-util.
  4. Forms fossa-deps.json file for all runtime packages.
  5. Performs fossa analyze and fossa test.

By default, FOSSA classifies this build as the following project and revision:

  • Project (in FOSSA): ${IMAGE_BASENAME}
  • Project Revision (in FOSSA): ${MACHINE}${IMAGE_VERSION_SUFFIX}

The project name and project revision can be customized by overriding the fossa analyze and fossa test commands. Please refer to Options for more details.

vendored-deps strategy

This strategy is activated by setting FOSSA_LICENSE_SCAN to "1"; for more detail on the difference see the FOSSA license scan section.

In this strategy, the fossa class does the following:

  1. Parses each recipe and persists JSON-ified recipe in tmp/fossa_metadata/recipes/.
  2. Copies the source code for each recipe to tmo/fossa_metadata/recipes/.
  3. Identifies runtime packages via oe.rootfs.image_list_installed_packages.
  4. Correlates runtime packages to recipe and package's metadata using a similar approach as scripts/oe-pkgdata-util.
  5. Forms fossa-deps.json file for all runtime packages.
  6. Performs fossa analyze and fossa test.

By default, FOSSA classifies this build as the following project and revision:

  • Project (in FOSSA): ${IMAGE_BASENAME}
  • Project Revision (in FOSSA): ${MACHINE}${IMAGE_VERSION_SUFFIX}

The project name and project revision can be customized by overriding the fossa analyze and fossa test commands. Please refer to Options for more details.

Options

These options are configured by adding the variable to build/conf/local.conf.

Variable Description Required Default
FOSSA_API_KEY The FOSSA API KEY. ✅ None
FOSSA_ENABLED Whether to run FOSSA. ❌ 1
FOSSA_TEST_ENABLED Whether to test the project for issues. ❌ 1
FOSSA_LICENSE_SCAN Whether to have FOSSA perform its own license scan of the dependencies (reference). ❌ 0
FOSSA_INIT_DEPS_JSON Path to a custom fossa-deps.json to be read during analysis (reference). ❌ None
FOSSA_CONFIG_FILE Path to fossa.yml (reference). ❌ None
FOSSA_EXCLUDE_PKGS_FROM_ANALYSIS Packages to exclude from final analysis (reference). ❌ None
FOSSA_RAW_ANALYZE_CMD Invoke fossa-cli with this argument string during the analysis step (reference). ❌ analyze --fossa-api-key <API_KEY> -p ${IMAGE_BASENAME} -r "${MACHINE}${IMAGE_VERSION_SUFFIX}"
FOSSA_RAW_TEST_CMD Invoke fossa-cli with this argument string during the test step (reference). ❌ test --fossa-api-key <API_KEY> -p ${IMAGE_BASENAME} -r "${MACHINE}${IMAGE_VERSION_SUFFIX}"
FOSSA_DEBUG Run fossa-cli in debug mode. Generally only required when troubleshooting. ❌ 0
FOSSA_OUTPUT_LOCAL Output the findings of fossa-cli locally instead of uploading to the FOSSA service. ❌ 0

Specify custom project name and project revision for FOSSA

Override invocation of analyze and test commands like the following:

FOSSA_RAW_ANALYZE_CMD = "analyze -p <PROJECT> -r <REVISION> --fossa-api-key <KEY>"
FOSSA_RAW_TEST_CMD = "test -p <PROJECT> -r <REVISION> --fossa-api-key <KEY>"

When these options are used, they must include the following data:

  • The FOSSA API key, via --fossa-api-key <KEY>.
  • The project name, via --project <PROJECT> or -p <PROJECT>.
  • The project revision, via --revision <REVISION> or -r <REVISION>.
  • The path to fossa.yml, if one is used, via --config <PATH> or -c <PATH>.

Typically meta-fossa provides all of these arguments as required, but it does not do so if FOSSA_RAW_ANALYZE_CMD or FOSSA_RAW_TEST_CMD are used.

If one command is customized, it is highly recommended to customize both; many options (and especially --fossa-api-key, --project, and --revision) should match between the commands.

For more information on available options, see the following documentation:

Manually include some dependencies in the analysis

There are scenarios in which it is desired to include additional dependencies in the analysis. This is supported with a fossa-deps.json file.

For example, assuming the following file at /home/user/example/fossa-deps-manual.json:

{
  "referenced-dependencies": [
    {
      "type": "gem",
      "name": "iron"
    },
    {
      "type": "pypi",
      "name": "Django",
      "version": "2.1.7"
    }
  ]
}

This path is then configured in build/conf/local.conf:

FOSSA_INIT_DEPS_JSON = "/home/user/example/fossa-deps-manual.json"

This results in the dependencies iron and Django being reported for the project, along with any other dependencies detected during the meta-fossa layer.

Exclude dependencies from the analysis

There are scenarios in which it is desired to omit reporting specific packages. This is supported via the FOSSA_EXCLUDE_PKGS_FROM_ANALYSIS option.

For example, to exclude bash and curl from the analysis, this option is configured in build/conf/local.conf:

FOSSA_EXCLUDE_PKGS_FROM_ANALYSIS = "bash curl"

This option is parsed as a list of space separated package names. Any detected package that matches an entry in this option, ignoring case, is omitted from the dependencies reported to FOSSA.

Perform only analysis (disregard fossa test)

fossa test blocks the build if the licenses for third party dependencies are incompatible with the policy set for this project in FOSSA.

For example, if the policy says "do not allow GPL code", and dependencies contain GPL code, fossa test blocks the build.

Some teams may not wish to block builds on FOSSA test results, either permanently or temporarily. This is supported via the FOSSA_TEST_ENABLED option.

This option is configured in build/conf/local.conf:

FOSSA_TEST_ENABLED = "0"

When set to 0, the meta-fossa layer does not block the build on the results of the FOSSA analysis.

Perform license scan

By default, fossa relies on the licenses provided in the build recipes when reporting dependencies. However, this relies on the entity creating the recipe to accurately report the license of the code built by the recipe. This is not always reliable.

In order to handle this case better, FOSSA can perform its own license scanning of dependencies instead of relying on the license reported by the recipe.

This option is configured in build/conf/local.conf:

FOSSA_LICENSE_SCAN = "1"

When set to 1, the meta-fossa layer performs its own license scan on recipe source code instead of relying on the license reported in the recipe.

Any patches are applied before inspecting the source for license data. No source code is uploaded to the FOSSA servers.

Internally, FOSSA treats the source code referenced in the recipe as "vendored dependencies"; for more information see the vendored dependencies feature documentation.

Important: Known issue

When using this option, if FOSSA has issues recursing into a dependency when scanning for licenses, that dependency is skipped and no error is raised, causing an incomplete license set to be uploaded to the FOSSA backend.

This will not affect all projects, but please check that your project is not affected prior to using this option. FOSSA is working on a patch for this issue and will update this repo when that fix is out.

Whether this affects your project can be determined by running a build with FOSSA_LICENSE_SCAN unset and checking the count of dependencies in FOSSA, then running a build with FOSSA_LICENSE_SCAN set. The count of dependencies should match, but if it does not then this issue affects your project.

To be notified of that fix, please contact support or create an issue on this repository asking to be notified when this issue is resolved.

Troubleshoot

If you are running into issues, please confirm the following:

I've included a layer in my image build

bitbake-layers show-layers

You should see meta-fossa in stdout.

I've inherited fossa in my image

Confirm by looking at conf/local.conf or relevant configurations.

I've set valid FOSSA_API_KEY

To verify, confirm that at least one of the following methods was used to specify the API key.

  • Via FOSSA_API_KEY
# e.g. bitbake -e core-image-minimal | grep "^FOSSA_API_KEY="
bitbake -e <IMAGE> | grep "^FOSSA_API_KEY="
  • Via FOSSA_RAW_ANALYZE_CMD AND FOSSA_RAW_TEST_CMD
bitbake -e <IMAGE> | grep "^FOSSA_RAW_ANALYZE_CMD="
bitbake -e <IMAGE> | grep "^FOSSA_RAW_TEST_CMD="
  • Via FOSSA_CONFIG_FILE
bitbake -e <IMAGE> | grep "^FOSSA_CONFIG_FILE="
# FOSSA_CONFIG_FILE=/some/path/fossa-config.yml

# >> cat /some/path/fossa-config.yml
# we should see the fossa-API-key set in the config file.

I can reach the fossa endpoint: https://app.fossa.com via curl

curl -vv https://app.fossa.com

If you are still running into an issue, please provide stdout of bitbake <image-base-name> -D, to the FOSSA support team, we can further investigate the issue for you.

FAQ

Why do meta-fossa recipes not cache?

meta-fossa recipes are set nostamp, so they always run.

This is required because in our testing we found that modifying patch sets does not cause the recipe to re-run, so we cannot rely on build system caching.

Why are vulnerabilities not supported?

FOSSA is actively developing and scoping a solution that will provide reliable and accurate vulnerability detection for yocto image builds.

Currently, we are in process of identifying how to correlate patch accurately, with CVE - and how to integrate within the FOSSA system, such that correction workflows, as well as issues workflows, work seamlessly.

How does FOSSA detect dependencies?

FOSSA only detects runtime dependencies of your yocto image build. It does so by looking at installed_packages on rootfs.

How can I report the defect?

You can report defect or feature request to the FOSSA team at https://support.fossa.com/hc/en-us

How does it differ from the create-spdx class?

The create-spdx class was introduced in Yocto with version 3.4 (honister), which does not work well for the current LTS release dunfell.

Further, meta-fossa is explicitly designed to work with FOSSA, which brings support for license correction workflows, org and project-wide reporting, notifications, and more.

How do I apply a correction to incorrect license or package information?

You can do so via FOSSA WEB UI. Please refer to the documentation at: https://docs.fossa.com/docs/triaging-issues#remediating-issues