helm used http proxy to access kube-apiserver when installing kubelet-csr-approver (with temporary solution) #11095

andrewfung729 · 2024-04-18T01:29:07Z

What happened?

Hi Kubespray Team. I am new to k8s. Glad to use kubespray!

I found that when installing kubelet-csr-approver, due to the lack of 'no_proxy' env, helm uses the http proxy to access kube-apiserver. Our corporate proxy logged the requests like CONNECT lb-apiserver.kubernetes.local:6443.

For the security and privacy, I have hidden the username, IPs, proxy URL.

Source shortcut: https://github.com/kubernetes-sigs/kubespray/blob/v2.24.1/roles/kubernetes-apps/kubelet-csr-approver/meta/main.yml

playbook log:

TASK [helm-apps : Update Helm repositories] ************************************************************************************************************************
task path: /home/user/kubespray/roles/helm-apps/tasks/main.yml:30
fatal: [cp-1]: FAILED! => {
    "changed": false,
    "command": "/usr/local/bin/helm list --output=yaml --filter dummy",
    "invocation": {
        "module_args": {
            "api_key": null,
            "atomic": false,
            "binary_path": "/usr/local/bin/helm",
            "ca_cert": null,
            "chart_ref": null,
            "chart_repo_url": null,
            "chart_version": null,
            "context": null,
            "create_namespace": false,
            "dependency_update": false,
            "disable_hook": false,
            "force": false,
            "history_max": null,
            "host": null,
            "kubeconfig": null,
            "post_renderer": null,
            "purge": true,
            "release_name": "dummy",
            "release_namespace": "kube-system",
            "release_state": "absent",
            "release_values": {},
            "replace": false,
            "set_values": null,
            "skip_crds": false,
            "state": "absent",
            "timeout": null,
            "update_repo_cache": true,
            "validate_certs": true,
            "values_files": [],
            "wait": false,
            "wait_timeout": null
        }
    },
    "msg": "Failure when executing Helm command. Exited 1.\nstdout: \nstderr: Error: Kubernetes cluster unreachable: Get \"https://lb-apiserver.kubernetes.local:6443/version\": tls: failed to verify certificate: x509: certificate signed by unknown authority\n",
    "stderr": "Error: Kubernetes cluster unreachable: Get \"https://lb-apiserver.kubernetes.local:6443/version\": tls: failed to verify certificate: x509: certificate signed by unknown authority\n",
    "stderr_lines": [
        "Error: Kubernetes cluster unreachable: Get \"https://lb-apiserver.kubernetes.local:6443/version\": tls: failed to verify certificate: x509: certificate signed by unknown authority"
    ],
    "stdout": "",
    "stdout_lines": []
}

What did you expect to happen?

It should be passed normally

TASK [helm-apps : Update Helm repositories] ************************************************************************************************************************
task path: /home/user/test/kubespray/roles/helm-apps/tasks/main.yml:6
ok: [cp-1] => {
    "changed": false,
    "command": "/usr/local/bin/helm",
    "invocation": {
        "module_args": {
            "api_key": null,
            "atomic": false,
            "binary_path": "/usr/local/bin/helm",
            "ca_cert": null,
            "chart_ref": null,
            "chart_repo_url": null,
            "chart_version": null,
            "context": null,
            "create_namespace": false,
            "dependency_update": false,
            "disable_hook": false,
            "force": false,
            "history_max": null,
            "host": null,
            "kubeconfig": null,
            "post_renderer": null,
            "purge": true,
            "release_name": "dummy",
            "release_namespace": "kube-system",
            "release_state": "absent",
            "release_values": {},
            "replace": false,
            "set_values": null,
            "skip_crds": false,
            "state": "absent",
            "timeout": null,
            "update_repo_cache": true,
            "validate_certs": true,
            "values_files": [],
            "wait": false,
            "wait_timeout": null
        }
    },
    "status": null,
    "stderr": "",
    "stderr_lines": [],
    "stdout": "",
    "stdout_lines": []
}

How can we reproduce it (as minimally and precisely as possible)?

Using kube-vip for HA (https://github.com/kubernetes-sigs/kubespray/blob/v2.24.1/docs/kube-vip.md)
Enable kubelet_csr_approver_enabled (hardening: https://github.com/kubernetes-sigs/kubespray/blob/v2.24.1/docs/hardening.md)

kubelet_rotate_server_certificates: true

Enable proxy (using proxy: https://github.com/kubernetes-sigs/kubespray/blob/v2.24.1/docs/proxy.md)

http_proxy: "http://foobar"
https_proxy: "http://foobar"
# https_proxy_cert_file: ""
# no_proxy: ""
# download_validate_certs: False
# additional_no_proxy: ""

OS

PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Version of Ansible

ansible [core 2.15.9]
config file = /home/user/test/kubespray/ansible.cfg
configured module search path = ['/home/user/test/kubespray/library']
ansible python module location = /home/user/miniconda3/envs/kubespray/lib/python3.11/site-packages/ansible
ansible collection location = /home/user/.ansible/collections:/usr/share/ansible/collections
executable location = /home/user/miniconda3/envs/kubespray/bin/ansible
python version = 3.11.8 (main, Feb 26 2024, 21:39:34) [GCC 11.2.0] (/home/user/miniconda3/envs/kubespray/bin/python)
jinja version = 3.1.2
libyaml = True

Version of Python

Python 3.11.8

Version of Kubespray (commit)

2cb8c85 (v2.24.1)

Network plugin used

calico

Full inventory with variables

provided in the reproduce method before

Command used to invoke ansible

ansible-playbook -i inventory/mycluster/hosts.yaml --become --become-user=root --ask-become-pass -vvvvvv cluster.yml

Output of ansible run

playbook output:

TASK [helm-apps : Update Helm repositories] ************************************************************************************************************************
task path: /home/user/kubespray/roles/helm-apps/tasks/main.yml:30
fatal: [cp-1]: FAILED! => {
    "changed": false,
    "command": "/usr/local/bin/helm list --output=yaml --filter dummy",
    "invocation": {
        "module_args": {
            "api_key": null,
            "atomic": false,
            "binary_path": "/usr/local/bin/helm",
            "ca_cert": null,
            "chart_ref": null,
            "chart_repo_url": null,
            "chart_version": null,
            "context": null,
            "create_namespace": false,
            "dependency_update": false,
            "disable_hook": false,
            "force": false,
            "history_max": null,
            "host": null,
            "kubeconfig": null,
            "post_renderer": null,
            "purge": true,
            "release_name": "dummy",
            "release_namespace": "kube-system",
            "release_state": "absent",
            "release_values": {},
            "replace": false,
            "set_values": null,
            "skip_crds": false,
            "state": "absent",
            "timeout": null,
            "update_repo_cache": true,
            "validate_certs": true,
            "values_files": [],
            "wait": false,
            "wait_timeout": null
        }
    },
    "msg": "Failure when executing Helm command. Exited 1.\nstdout: \nstderr: Error: Kubernetes cluster unreachable: Get \"https://lb-apiserver.kubernetes.local:6443/version\": tls: failed to verify certificate: x509: certificate signed by unknown authority\n",
    "stderr": "Error: Kubernetes cluster unreachable: Get \"https://lb-apiserver.kubernetes.local:6443/version\": tls: failed to verify certificate: x509: certificate signed by unknown authority\n",
    "stderr_lines": [
        "Error: Kubernetes cluster unreachable: Get \"https://lb-apiserver.kubernetes.local:6443/version\": tls: failed to verify certificate: x509: certificate signed by unknown authority"
    ],
    "stdout": "",
    "stdout_lines": []
}

verbose output show the proxy env

<10.XX.XX.XX> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/user/.ssh/user-ansible"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="user"' -o ConnectTimeout=10 -o 'ControlPath="/home/user/.ansible/cp/2d48a40eb9"'10.XX.XX.XX '/bin/sh -c '"'"'sudo -H -S -n  -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-hrfrnpvqtsxvhxtnqdnfjewoftnqwsdo ; ALL_PROXY='"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"''"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"' FTP_PROXY='"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"''"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"' HTTPS_PROXY='"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"''"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"' HTTP_PROXY='"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"''"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"' NO_PROXY='"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"''"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"' all_proxy='"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"''"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"' ftp_proxy='"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"''"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"' http_proxy=http://foobar https_proxy=http://foobar no_proxy='"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"''"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"' /usr/bin/python3'"'"'"'"'"'"'"'"' && sleep 0'"'"''
Escalation succeeded
<10.XX.XX.XX> (1, b'\n{"stdout": "", "stderr": "Error: Kubernetes cluster unreachable: Get \\"https://lb-apiserver.kubernetes.local:6443/version\\": tls: failed to verify certificate: x509: certificate signed by unknown authority\\n", "command": "/usr/local/bin/helm list --output=yaml --filter kubelet-csr-approver", "failed": true, "msg": "Failure when executing Helm command. Exited 1.\\nstdout: \\nstderr: Error: Kubernetes cluster unreachable: Get \\"https://lb-apiserver.kubernetes.local:6443/version\\": tls: failed to verify certificate: x509: certificate signed by unknown authority\\n", "invocation": {"module_args": {"atomic": true, "binary_path": "/usr/local/bin/helm", "name": "kubelet-csr-approver", "namespace": "kube-system", "chart_ref": "kubelet-csr-approver/kubelet-csr-approver", "chart_version": "0.2.8", "wait": true, "values": {}, "release_name": "kubelet-csr-approver", "release_namespace": "kube-system", "release_values": {}, "validate_certs": true, "dependency_update": false, "release_state": "present", "values_files": [], "update_repo_cache": false, "disable_hook": false, "force": false, "purge": true, "create_namespace": false, "replace": false, "skip_crds": false, "context": null, "kubeconfig": null, "host": null, "ca_cert": null, "api_key": null, "chart_repo_url": null, "wait_timeout": null, "timeout": null, "post_renderer": null, "history_max": null, "set_values": null}}}\n', b"OpenSSH_8.9p1 Ubuntu-3ubuntu0.6, OpenSSL 3.0.2 15 Mar 2022\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 10.XX.XX.XX is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/user/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/user/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 418983\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 1\r\n")

Anything else we need to know

My solution is adding back no_proxy env (https://github.com/kubernetes-sigs/kubespray/blob/v2.24.1/roles/kubernetes-apps/kubelet-csr-approver/meta/main.yml)

---
dependencies:
  - role: helm-apps
    when:
      - inventory_hostname == groups['kube_control_plane'][0]
      - kubelet_csr_approver_enabled
    environment:
      http_proxy: "{{ http_proxy | default('') }}"
      https_proxy: "{{ https_proxy | default('') }}"
      no_proxy: "{{ no_proxy | default('') }}" # <-------  add this
    release_common_opts: {}
    releases:
      - name: kubelet-csr-approver
        namespace: "{{ kubelet_csr_approver_namespace }}"
        chart_ref: "{{ kubelet_csr_approver_chart_ref }}"
        chart_version: "{{ kubelet_csr_approver_chart_version }}"
        wait: true
        values: "{{ kubelet_csr_approver_values }}"
    repositories:
      - name: "{{ kubelet_csr_approver_repository_name }}"
        url: "{{ kubelet_csr_approver_repository_url }}"

I also found a similar structure at custom_cni (https://github.com/kubernetes-sigs/kubespray/blob/v2.24.1/roles/network_plugin/custom_cni/meta/main.yml)

The text was updated successfully, but these errors were encountered:

KubeKyrie · 2024-05-08T13:09:23Z

Could u show the kube-vip setting?

andrewfung729 · 2024-05-09T06:02:13Z

Thanks for your reply. For your information, after I added back the no_proxy env on playbook, the issue no longer occurred again.

## `inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml`
kube_proxy_strict_arp: true

## `inventory/mycluster/group_vars/k8s_cluster/addons.yml`
# Kube VIP
kube_vip_enabled: true
kube_vip_arp_enabled: true
kube_vip_controlplane_enabled: true
kube_vip_address: 10.XX.XX.XX
loadbalancer_apiserver:
  address: "{{ kube_vip_address }}"
  port: 6443
kube_vip_interface: enp6s19
kube_vip_lb_enable: true
kube_vip_services_enabled: false
kube_vip_enableServicesElection: true

andrewfung729 · 2024-06-13T05:47:09Z

@KubeKyrie I think it just needs a simple fix no_proxy: "{{ no_proxy | default('') }}". Can I make a pull request for that?

FYI, to enable kubelet-csr-approver we need kubelet_rotate_server_certificates: true.

k8s-triage-robot · 2024-09-11T06:39:30Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

andrewfung729 · 2024-09-11T06:41:58Z

/remove-lifecycle stale

k8s-triage-robot · 2024-12-10T07:16:00Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

andrewfung729 added the kind/bug Categorizes issue or PR as related to a bug. label Apr 18, 2024

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 11, 2024

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 11, 2024

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Dec 10, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

helm used http proxy to access kube-apiserver when installing kubelet-csr-approver (with temporary solution) #11095

helm used http proxy to access kube-apiserver when installing kubelet-csr-approver (with temporary solution) #11095

andrewfung729 commented Apr 18, 2024

KubeKyrie commented May 8, 2024

andrewfung729 commented May 9, 2024 •

edited

Loading

andrewfung729 commented Jun 13, 2024 •

edited

Loading

k8s-triage-robot commented Sep 11, 2024

andrewfung729 commented Sep 11, 2024

k8s-triage-robot commented Dec 10, 2024

helm used http proxy to access kube-apiserver when installing kubelet-csr-approver (with temporary solution) #11095

helm used http proxy to access kube-apiserver when installing kubelet-csr-approver (with temporary solution) #11095

Comments

andrewfung729 commented Apr 18, 2024

What happened?

What did you expect to happen?

How can we reproduce it (as minimally and precisely as possible)?

OS

Version of Ansible

Version of Python

Version of Kubespray (commit)

Network plugin used

Full inventory with variables

Command used to invoke ansible

Output of ansible run

Anything else we need to know

KubeKyrie commented May 8, 2024

andrewfung729 commented May 9, 2024 • edited Loading

andrewfung729 commented Jun 13, 2024 • edited Loading

k8s-triage-robot commented Sep 11, 2024

andrewfung729 commented Sep 11, 2024

k8s-triage-robot commented Dec 10, 2024

andrewfung729 commented May 9, 2024 •

edited

Loading

andrewfung729 commented Jun 13, 2024 •

edited

Loading