undo node that has been marked for deletion

[devopsmash]

Description

I have a scenario that I forgot to replace expireAfter: 720h to expireAfter: Never in some critical nodes that host important pods (like databases).

luckily, I have the annotation karpenter.sh/do-not-disrupt: "true" on my important pods which prevent the node deletion.

After changing the NodePool to expireAfter: Never the Karpenter still tries to remove my node. why?

Even if I try to remove the taint karpenter.sh/disrupted:NoSchedul

kubectl taint node <mynode> karpenter.sh/disrupted:NoSchedule-

and the relevant annotations

deletionGracePeriodSeconds: 0
deletionTimestamp: '2024-11-26T13:01:03Z'

Somehow the taint and the annotations above returned immediately and the node is still in Removing state.

Any idea how to undo node that has been marked for deletion?
Not sure if this is a new Feature or a question

[engedaam]

For v1.0, karpenter is now using forced expiration on nodes, therefore the only way to reset expiration is to recycle all your nodes. We have a design document that outlines our thinking: https://github.com/kubernetes-sigs/karpenter/blob/main/designs/forceful-expiration.md .There is also a feature request to support the idea of using graceful expiration: kubernetes-sigs/karpenter#1750. Would it be okay to close out this issue in favor of the issue mentioned?

[devopsmash]

Thank you @engedaam for your response.

Just to clarify, there is no way (even manually) to interfere Karpenter and undo node removal that has been marked for deletion?
This means that I have to delete this node eventually?