Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Avoid invoking build multiple times in CI pipeline #2589

Open
nijel opened this issue Aug 26, 2024 · 4 comments
Open

Avoid invoking build multiple times in CI pipeline #2589

nijel opened this issue Aug 26, 2024 · 4 comments
Labels
enhancement Adding or requesting a new feature. good first issue Opportunity for newcoming contributors. hacktoberfest This is suitable for Hacktoberfest. Don’t try to spam. help wanted Extra attention is needed.

Comments

@nijel
Copy link
Member

nijel commented Aug 26, 2024 •
edited
Loading

Describe the problem

Right now, we're misusing Docker caching between CI build steps to share the built image. This is fragile as can easily lead to building a different image in each job without noticing.

Instead, the build step should produce the images which would be tested, merged and published, where additional steps would not invoke build.

Describe the solution you would like

There is documentation to follow on this topic: https://docs.docker.com/build/ci/github-actions/multi-platform/#distribute-build-across-multiple-runners

https://docs.docker.com/build/ci/github-actions/share-image-jobs/ might also be useful, but that probably won't work with multi arch images (but might work for testing amd64 images)

Describe alternatives you have considered

No response

Screenshots

No response

Additional context

No response
@nijel nijel added enhancement Adding or requesting a new feature. help wanted Extra attention is needed. hacktoberfest This is suitable for Hacktoberfest. Don’t try to spam. good first issue Opportunity for newcoming contributors. labels Aug 26, 2024
@github-actions GitHub Actions
Copy link

This issue seems to be a good fit for newbie contributors. You are welcome to contribute to Weblate! Don't hesitate to ask any questions you would have while implementing this.

You can learn about how to get started in our contributors documentation.

nijel added a commit to nijel/docker that referenced this issue Aug 26, 2024
@nijel
feat(ci): add merge step to CI pipeline
8d48dcd 
- this verifies that merging actually works before doing publish
- can be later used for proper merge as described in WeblateOrg#2589
@nijel nijel mentioned this issue Aug 26, 2024
Merged
5 tasks
nijel added a commit that referenced this issue Aug 26, 2024
@nijel
feat(ci): add merge step to CI pipeline
f5b00f9 
- this verifies that merging actually works before doing publish
- can be later used for proper merge as described in #2589
nijel added a commit to nijel/docker that referenced this issue Sep 10, 2024
@nijel
feat(ci): push built images to ghcr.io registry
546497a 
This is a very small step towards WeblateOrg#2589.
@nijel nijel mentioned this issue Sep 10, 2024
Closed
5 tasks
@nijel
Copy link
Member Author

nijel commented Sep 10, 2024

As shown in #2632, pushing won't work from pull requests (we would rather not grant access to images registry to untrusted builds). So probably the only usable approach to this is to use artifacts, but the question is how to make it work with multi-architecture images we need to produce.

@nijel
Copy link
Member Author

nijel commented Nov 1, 2024

Reverting afa5c4c might be a way to go, just use artifacts instead of caches.

nijel added a commit to nijel/docker that referenced this issue Nov 5, 2024
@nijel
feat: split out docker build
376ef7e 
- use base images built separately that contain all system dependencies
- use multistage build to avoid devel deps in the final image
- this also reduces pressure on GitHub cache, see WeblateOrg#2589
nijel added a commit to nijel/docker that referenced this issue Nov 5, 2024
@nijel
feat: split out docker build
aac4435 
- use base images built separately that contain all system dependencies
- use multistage build to avoid devel deps in the final image
- this also reduces pressure on GitHub cache, see WeblateOrg#2589
nijel added a commit to nijel/docker that referenced this issue Nov 5, 2024
@nijel
feat: split out docker build
2f3c673 
- use base images built separately that contain all system dependencies
- use multistage build to avoid devel deps in the final image
- this also reduces pressure on GitHub cache, see WeblateOrg#2589
nijel added a commit to nijel/docker that referenced this issue Nov 5, 2024
@nijel
feat: split out docker build
e49daba 
- use base images built separately that contain all system dependencies
- use multistage build to avoid devel deps in the final image
- this also reduces pressure on GitHub cache, see WeblateOrg#2589
nijel added a commit that referenced this issue Nov 5, 2024
@nijel
feat: split out docker build
5bf9347 
- use base images built separately that contain all system dependencies
- use multistage build to avoid devel deps in the final image
- this also reduces pressure on GitHub cache, see #2589
@nijel
Copy link
Member Author

nijel commented Nov 8, 2024

Okay, this is really tricky. I've tried to make this work at WeblateOrg/docker-base#4 and there are several obstacles in Docker:

  • loading/saving works for images, not manifests
  • there is no way to build source manifest for an image locally
  • all image merging approaches are based on pushing to the registry what we don't want for pull requests

@nijel nijel mentioned this issue Nov 8, 2024
Closed
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement Adding or requesting a new feature. good first issue Opportunity for newcoming contributors. hacktoberfest This is suitable for Hacktoberfest. Don’t try to spam. help wanted Extra attention is needed.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@nijel