entrypoint.sh not being run after pod is recreated

[Eneuman]

Describe the bug
If I delete a pod in my Dev Spaces cluster it gets recreated but the service does not start correctly so I receive a "Bad Gateway" error in my web browser.

To recreate:

1. Create a .Net Core 3.1 Gui Service.
2. Start it by pressing F5 in VS.
3. Start K8 Dashboard, navigate to the pod and click "Delete".
4. Watch as the pod is being recreated and enters running state.
5. Try and access your application using a web browser.
6. You will now receive "Bad Gateway"

If I execute the following command:

kubectl exec "My pod name" -- dotnet run --no-build --no-launch-profile -c Debug

The application starts fine and I can access it again.

It seems like the scheduler in K8 does not start the pod correctly if it's being deleted by the dashboard.

The same behavior occurs if a pod is being moved from one node to another in a multi node Dev Space environment.

[sanar-microsoft]

@Eneuman Thanks for pointing out this issue, just to be sure that there is no network problem on the cluster can you try recreating(delete) the pods in kube-system and azds namespace, especially traefik, coredns and kube-proxy pods. Also to rule out any issue with the Dockerfile.develop, do you see the same behavior when you do azds up through the CLI?

I've run into a similar issue before due to a network issue on the cluster and after restarting pods in kube-system namespace and doing azds down before running F5 again fixed the issue.

[Eneuman]

Hi,
I tried your suggestions but it didn't help :(
The entrypoint.sh is not being run.

(I am running Dev Spaces in a 2 node cluster)

Here are some info about the pod I just tried this with (after it was recreated)

[sanar-microsoft]

@Eneuman Were you able to try azds down and azds up on the service? Are you able to hit the endpoint then? A pod created through azds up will continue to work even if you recreate it by deleting the pod.
If you do F5 again on this pod, do things work or do you still get Bad Gateway? If you delete this pod, the new pod that comes up will not have the application running unless you do an F5 again.

During F5 scenario the application is started by another process after receiving a request from the client(VS), my guess is that the call never reached the running process in the pod and the actual application is therefore not running.

[Eneuman]

Hi
Yeah, after pressing F5 the app starts. But if I delete the pod, it does not start the app and I receive a bad gateway error.

All our pods are created using F5 because of our development model. But since we are using 2 nodes in our cluster, the master nodes moves our services around automatically and this makes them stop working. Since we have 36 micro services, we need to manually start the moved services each day :(.

Is it possible to make a pod, started by F5, to also start the app when it gets recreated by the scheduler?

[sanar-microsoft]

Our current implementation doesn't support it, but I've opened a work item on the team to address it.
For services that don't require debugging, running them through azds up instead of F5 can keep the pods running even if they are moved between the nodes.