Can not create container when systemd-resolved not running

[vaclavskala]

What happened?

Playbook cluster.yml crash on Kubeadm | Create kubeadm config when systemd-resolved is not running because /run/systemd/resolve/resolv.conf file is missing.

What did you expect to happen?

Kubespray will configure kubelet to use /etc/resolv.conf instead of missing /run/systemd/resolve/resolv.conf,

How can we reproduce it (as minimally and precisely as possible)?

Run cluster.yml on kube nodes running Ubuntu 24.04 with systemd-resolved masked.

OS

Linux 6.1.113-zfs226 x86_64
PRETTY_NAME="Ubuntu 24.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.1 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

Version of Ansible

ansible [core 2.16.14]
  config file = /var/home/kubespray-2.26.0/ansible.cfg
  configured module search path = ['/var/home/kubespray-2.26.0/library']
  ansible python module location = /var/home/kubespray-2.26.0/venv/lib/python3.12/site-packages/ansible
  ansible collection location = /var/home/ansible/collections:/usr/share/ansible/collections
  executable location = /var/home/kubespray-2.26.0/venv/bin/ansible
  python version = 3.12.3 (main, Nov  6 2024, 18:32:19) [GCC 13.2.0] (/var/home/kubespray-2.26.0/venv/bin/python)
  jinja version = 3.1.4
  libyaml = True

Version of Python

Python 3.12.3

Version of Kubespray (commit)

kubespray-2.26.0

Network plugin used

calico

Full inventory with variables

Default kubespray-2.26.0 variables

Command used to invoke ansible

ansible-playbook -i inventory/cluster/inventory.ini cluster.yml

Output of ansible run

TASK [kubernetes/control-plane : Kubeadm | Create kubeadm config] **************************************************************************************************************************************************************************
changed: [XXX-prod-master1]
changed: [XXX-prod-master2]
changed: [XXX-prod-master3]
Tuesday 17 December 2024  12:38:03 +0100 (0:00:00.492)       0:07:22.794 ****** 
Tuesday 17 December 2024  12:38:03 +0100 (0:00:00.043)       0:07:22.837 ****** 
Tuesday 17 December 2024  12:38:03 +0100 (0:00:00.047)       0:07:22.885 ****** 
Tuesday 17 December 2024  12:38:03 +0100 (0:00:00.041)       0:07:22.927 ****** 
Tuesday 17 December 2024  12:38:03 +0100 (0:00:00.048)       0:07:22.976 ****** 
Tuesday 17 December 2024  12:38:04 +0100 (0:00:00.090)       0:07:23.067 ****** 
Tuesday 17 December 2024  12:38:04 +0100 (0:00:00.100)       0:07:23.167 ****** 
Tuesday 17 December 2024  12:38:04 +0100 (0:00:00.054)       0:07:23.221 ****** 
Tuesday 17 December 2024  12:38:04 +0100 (0:00:00.044)       0:07:23.266 ****** 
Tuesday 17 December 2024  12:38:04 +0100 (0:00:00.048)       0:07:23.315 ****** 
Tuesday 17 December 2024  12:38:04 +0100 (0:00:00.055)       0:07:23.370 ****** 
FAILED - RETRYING: [XXX-prod-master1]: Kubeadm | Initialize first master (3 retries left).
FAILED - RETRYING: [XXX-prod-master1]: Kubeadm | Initialize first master (2 retries left).
FAILED - RETRYING: [XXX-prod-master1]: Kubeadm | Initialize first master (1 retries left).

Anything else we need to know

Problem is that in roles/kubernetes/preinstall/tasks/main.yml there is detection if systemd-resolved is running but it is only used to detect if include 0060-resolvconf.yml or 0061-systemd-resolved.yml.

But in roles/kubernetes/node/tasks/facts.yml is included OS specific var file from roles/kubernetes/node/vars and in that file resolvconf path is hardcoded for most distributions to /run/systemd/resolve/resolv.conf.
And it cause kubelet fail to create any container.

On control-plane servers this cause kubelet can not create any container with error:
Dec 17 14:01:15 XXX-prod-master1 kubelet[25126]: E1217 14:01:15.267321   25126 dns.go:284] "Could not open resolv conf file." err="open /run/systemd/resolve/resolv.conf: no such file or directory"
Dec 17 14:01:15 XXX-prod-master1 kubelet[25126]: E1217 14:01:15.267332   25126 kuberuntime_sandbox.go:45] "Failed to generate sandbox config for pod" err="open /run/systemd/resolve/resolv.conf: no such file or directory" pod="kube-system/kube-controller-manager-XXX-prod-master1"
Dec 17 14:01:15 XXX-prod-master1 kubelet[25126]: E1217 14:01:15.267342   25126 kuberuntime_manager.go:1166] "CreatePodSandbox for pod failed" err="open /run/systemd/resolve/resolv.conf: no such file or directory" pod="kube-system/kube-controller-manager-XXX-prod-master1"
Dec 17 14:01:15 XXX-prod-master1 kubelet[25126]: E1217 14:01:15.267361   25126 pod_workers.go:1298] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"kube-controller-manager-XXX-prod-master1_kube-system(bce3ce42e0aef110c5773ef4027de42c)\" with CreatePodSandboxError: \"Failed to generate sandbox config for pod \\\"kube-controller-manager-XXX-prod-master1_kube-system(bce3ce42e0aef110c5773ef4027de42c)\\\": open /run/systemd/resolve/resolv.conf: no such file or directory\"" pod="kube-system/kube-controller-manager-XXX-prod-master1" podUID="bce3ce42e0aef110c5773ef4027de42c"

When systemd-resolved is not running on worker nodes, any container is stuck in ContainerCreating state with error:

Events:
  Type     Reason                  Age                   From               Message
  ----     ------                  ----                  ----               -------
  Normal   Scheduled               5m52s                 default-scheduler  Successfully assigned kube-system/kube-proxy-6hnnc to XXX-prod-worker2
  Warning  FailedCreatePodSandBox  44s (x26 over 5m52s)  kubelet            Failed to create pod sandbox: open /run/systemd/resolve/resolv.conf: no such file or directory