Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Additional exportable layers #181

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

Additional exportable layers #181

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
bef0d58
Additional exportable layers
sambhav Aug 13, 2021
b56ed25
Add alternative proposal
sambhav Sep 16, 2021
eda604b
Update the RFC to use multiple app dirs idea
sambhav Dec 1, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 86 additions & 0 deletions text/0000-additional-exported-layers.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
# Meta
[meta]: #meta
- Name: Additional exportable layers
- Start Date: 2021-08-13
- Author(s): [@samj1912](https://github.com/samj1912)
- RFC Pull Request: (leave blank)
- CNB Pull Request: (leave blank)
- CNB Issue: (leave blank)
- Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC)

# Summary
[summary]: #summary

Allow users to provide additional volumes that can be exported similar to the application directory and allow easy configuration of the application directory. These volumes are defined via the `buildpack.exportable-volumes` by each buildpack.

# Definitions
[definitions]: #definitions

- `exportable-volumes`: A set of volumes describing where the buildpack process is likely to write data specific and should be exported alongside the application workspace.

# Motivation
[motivation]: #motivation

The main motivation behind this RFC is to unlock exporting application images that require libraries or software to be installed in certain directories apart from the default application directory and these installations are rebase safe and self-contained. This unlocks use cases where buildpacks may install software in `/opt` directories (for AWS Lambda extensions or other common standalone software) for example or preserve some settings in a `/home/$USER/.config` for the output application. This provides a rebase safe extension point without major changes to the buildpacks API.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the motivation is to allow for "libraries or software that install in certain directories", it goes at odds with requiring that "these exportable volumes be empty". After all, we can't guarantee that these "libraries or software" aren't installing to / or something.


# What it is
[what-it-is]: #what-it-is

Buildpack authors would be able to buildpacks with `buildpack.exportable-volumes` keys in their `buildpack.toml` files.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Buildpack authors would be able to buildpacks with `buildpack.exportable-volumes` keys in their `buildpack.toml` files.
Buildpack authors would be able define a set of `buildpack.exportable-volumes` keys in their `buildpack.toml` files.


For example a a buildpack may look like -

```toml
api = "<buildpack API version>"

[buildpack]
id = "<buildpack ID>"
name = "<buildpack name>"
exportable-volumes = ["aws-extensions", "user-config-dir"]
```

The above would only be valid for normal buildpacks and not meta-buildpacks. Based on the `exportable-volumes` field, a platform would be responsible for validating that the `build` and `run` image do not have any content on these paths. By default these volume names will map to `/workspaces/<exportable-volume-name>` but a user may specify a label `io.buildpacks.exportable-volumes` on the `build` image to map a volume name to a different location. This label would look like -
jkutner marked this conversation as resolved.
Show resolved Hide resolved

```json
[
{"name": "user-config-dir", "value": "/home/cnb/.config"},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this mapping concept being done by the base image author.

Could this list be something buildpack authors could get at and then name layers that match user-config-dir that gets exported to /home/cnb/.config during export phase? Or perhaps a key in the layer.toml has to be set so it is more explicit.

The requirement for an empty dir and symlinking for builder could potentially be avoided. A buildpack author could put a new binary in the resulting app image's /usr/bin by putting the binary in /layers/<bp-dir>/usr-bin/. If a new stack is used that doesn't support it, the layer would still exist on the resulting app image but not in the overridden path.

Giving the buildpack a dictionary so they can easily check if "usr/bin" is available to use would allow buildpack authors to either put the binary in "usr/bin" or do more work like altering PATH and whatnot to get it to work on other base images.

{
  "usr/bin" : "usr-bin",
  "~/.config": "user-config-dir"
}

{"name": "aws-extensions", "value": "/opt"}
]
```

For volumes defined by buildpacks but not in the `io.buildpacks.exportable-volumes` we will use a default `/workspaces/<exportable-volume-name>` directory.

# How it Works
[how-it-works]: #how-it-works

This RFC would require changes to the lifecycle and platform API.

The lifecycle would be responsible for creating symlinks from `/workspaces/<exportable-volume-name>` in the build image to the actual mapped directory and make these writable for buildpacks. It will also be responsible for exporting the content on these directories to the actual paths on the run-image.

The current default workspace directory would be moved to `/workspaces/default` but the lifecycle will symlink that directory back to `/workspace` for backwards compatibility.

# Drawbacks
[drawbacks]: #drawbacks

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be hard for a simple platform like tekton to implement this.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible this could open a new attack vector? I can't think of anything specific, but i wonder what's possible if a buildpack can drop files in /etc/ or /usr/bin/ for example.

- Additional complexity

# Alternatives
[alternatives]: #alternatives

N/A
# Prior Art
[prior-art]: #prior-art

- [RFC 72](https://github.com/buildpacks/rfcs/blob/main/text/0072-image-workdir.md)
- [RFC PR #172](https://github.com/buildpacks/rfcs/pull/172)
- [RFC PR #173](https://github.com/buildpacks/rfcs/pull/173)

# Unresolved Questions
[unresolved-questions]: #unresolved-questions

TBD.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there any security ramifications for certain paths? Do platforms or builders need a way to deny paths?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be good to have a block list of paths.


# Spec. Changes (OPTIONAL)
[spec-changes]: #spec-changes

As noted above.