Add CI/CD for building and pushing docker image to the public registry on release

[yevhen-kalyna]

Is your feature request related to a problem? Please describe.
For using GDG in docker I should build and store images by myself

Describe the solution you'd like
Better to create a public docker registry with images by versions in it, it will be very useful for fast CLI usage via docker, and in helm charts/k8s clusters. This repository already has CI/CD for releasing new versions and a pretty good docker image, let's add a new docker release target in CI/CD process by building and pushing the image to the public docker registry (GitHub, DockerHub)

[yevhen-kalyna]

@safaci2000 Hello! Please review my issue and PR. Also, before the merge, you should create a DockerHub registry and fill up secrets for the pipeline.

[safaci2000]

@yevhen-kalyna

I don't mind having the image being published, but what use case do you see for gdg running in a helm chart or kubernetes? It's a pretty standard alone app.

As far as publishing docker containers. I think it would be better to use https://goreleaser.com/customization/docker/ since it's the tool responsible for building all the release artifacts, it makes more sense to have it do the work than having another workflow.

Need to figure out where to drop it. Technically esnet has a public registry to be used for OSS project but needs some work to get it in a usable state. Alternatively github can act as a docker registry as well.

[yevhen-kalyna]

@safaci2000
In my case I have 5 different k8s clusters in 4 different AWS accounts, each cluster contains Grafana + 3 data sources. In one Grafana instance, I will edit dashboards, and with GDG (in docker image, helm chart installed in a cluster) I will import the dashboard and export them to other instances every N minutes. Previously I used Grafana Wizzy, but this tool does not support new versions of Grafana and is not maintained anymore.

I just suggest the first solution comlex solution. You can choose whatever you think will be better, I just need GDG in docker image with public registry.

BTW, I guess people who recreate a lot of VMs will prefer to use GDG in docker, better than installing Go, dependencies and etc

[safaci2000]

Well, you don't need to install Go to run GDG, that's part of the appeal of go, is the output is a statically compiled binary. You can find them in the release page. That being said, having a public docker container should be easy enough. Just need to get some time to figure out where to push, thanks for the PR though. I appreciate the effort.

[safaci2000]

@yevhen-kalyna I added a CICD pattern to publish a docker container. Can you confirm that you can pull the following images:

ghcr.io/esnet/gdg:latest
ghcr.io/esnet/gdg:v0.3.1
ghcr.io/esnet/gdg:0.3

More details can be found on here: https://github.com/esnet/gdg/pkgs/container/gdg

[yevhen-kalyna]

@safaci2000

$ docker pull ghcr.io/esnet/gdg:v0.3.1
Error response from daemon: Head "https://ghcr.io/v2/esnet/gdg/manifests/v0.3.1": denied: denied

Same for all images. Also, the link shows 404 error. I guess you created a private registry.

[yevhen-kalyna]

BTW, gdg in docker container can't create any folders and files, lack of permissions for user.

�[31mPANI�[0m[0000] unable to create path prod/dashboards/General  �[31merror�[0m="mkdir prod: permission denied"
panic: (*logrus.Entry) 0xc00013e1c0

[safaci2000]

I need an override from the powers the org level to make the package public. Working on that, for now, pull this: safaci2000/testing:gdg_0_3_1 It should be the same image. The image is running as root so there really shouldn't be an issue with creating folders.

There are two volumes defined:
- /app/conf
-/app/exports

you should be able to override both and set them to anything local volume of K8s storage.

[safaci2000]

This should all be resolved. Let me know when you have a chance. Should be able to pull from the repo directly.

[yevhen-kalyna]

@safaci2000, Sorry for the late response. Just checked, and got the same error.

$ docker pull ghcr.io/esnet/gdg:v0.3.1
Error response from daemon: Head "https://ghcr.io/v2/esnet/gdg/manifests/v0.3.1": denied: denied

Registry not public. For testing purposes, you can docker logout ghcr.io and test if an image can be pulled or not.

[safaci2000]

I had some issue with the versioning where the v was included or excluded in some cases. Try pulling:

docker pull ghcr.io/esnet/gdg:0.3.1 or docker pull ghcr.io/esnet/gdg:0.3 both of which should give you the same docker binary.

Let me know if you still have issues. I confirmed the image is public from a GCP VM so you should be good to go.

[yevhen-kalyna]

You're right, it's my local issue with pulling from GitHub. Will fix it and come to you with feedback about image. Thanks!

[yevhen-kalyna]

I pulled the image end tested it with k8s. But see a few issues with docker image and gdg:

1. Dockerfile does not contain WORKDIR /app, because of this, config folder /app/config is invisible for gdg and tool will create config folder in root, like /conf. I fixed it in my k8s manifest, but FYI.
2. Run gdg with -c or --config flag will be ignored by tool. If I move config to /conf it will work, but not with -c flag. Example:

/app/gdg dash list --config /app/conf/importer.yml
WARN[0000] Error getting organizations: Get "https://staging.grafana.com/api/orgs": dial tcp: lookup staging.grafana.com on 192.168.0.1:53: no such host

[safaci2000]

docker run -it --rm --entrypoint /bin/sh ghcr.io/esnet/gdg:0.3.1 The config folder is named conf it won't search for configurations stored in the config folder by default by default. It also should accept anything as a CMD argument. The entry point is set to /app/gdg

#find /app/
/app/
/app/exports
/app/conf
/app/gdg

it has a really basic default config embedded into the binary but if you simply have a volume mount that has a YML file loaded in /app/conf/importer.yml for example it should work fine.

At this point this is more of a discussion, I'm going to convert this to a conversation thread as I believe this is working as expected from my perspective.

[safaci2000]

@yevhen-kalyna Try pulling the same tags again. I was missing the WORKDIR directive so everything was running off the root of the image. It should work now. Or add a working_dir: /app directive to your docker-compose manifest or equivalent for K8.

[yevhen-kalyna]

Working for me, working_dir fixed it from scratch. Thanks.

[safaci2000]

Awesome. Enjoy and let me know if you run into any more trouble. I added some minor documentation on here as well: https://software.es.net/gdg/docs/installation/#docker-usage