Questions regarding the server node's agent-token parameter

[ehlesp]

Although the agent-token parameter (available in a server node's configuration) is currently experimental, I think it would be helpful to have some explanation about the following.

1. In a multiserver cluster setup, has the agent-token parameter to be specified on all the server nodes' configuration?
   • Wouldn't be enough if only the first server node created defines the agent-token's value for the whole cluster?
2. Also in a multiserver setup, can the value be different on each server on the same cluster, or is there some validation of the value done among the servers? My take on this is that it wouldn't make sense to have a different agent-token defined on each server node of the same cluster, but since this is not documented I cannot really tell.
3. Can any string be passed as agent-token or it has to comply with a certain format or limitations, like minimum and maximum lengths? I'm asking this because I happened to have trouble using a string formatted somewhat like a server token.
   • The first string I defined as agent-token in the two server nodes of a cluster was K1024oqngb2xopj35k9wb877bb56objobkj2874dz28hqfq9n74gwd7vg5273p9r6354::agent::dz6k5nssgmu8fko7h559pde82q8howgf.
   • When any agent node tried to connect to the cluster with that string as token, in the log of those agents I would see the error line level=error msg="invalid token CA hash length".
   • Then I reinstalled the servers and set a different agent-token, which was similarly formatted but just one character longer than the previous one, and then the agents were able to connect to the cluster.
4. After a successful installation of a K3s cluster, can the value of agent-token be changed just by editing it in the config.yaml or k3s.service files of the server nodes, or is there something else to do (maybe changing some register in the etcd db)?

[ehlesp]

I ended up testing the behaviour of the agent-token parameter myself with the v1.21.1+k3s1 version of K3s, so here are my takes from what I've seen.

1. The agent-token is a per-server node parameter, meaning that its neither propagated nor compared among the other server nodes that might join the cluster later. Therefore, each server node can have its own particular agent-token, so an agent node must use the corresponding correct token when accessing through a particular server node.
2. The agent-token can be changed directly on whichever file it was left configured within the server node, either in the config.yaml or in the k3s.service script. It doesn't seem to be registered anywhere else. Of course, this has to be done with care since agent nodes may be accessing the cluster with the changed token.

The only question that remains open for me is about the correct format of the token itself. As I've pointed out in my initial post, I've meet with some strange problem that seems to be related with this detail. I was able to get around it just by generating another token, but I think that a clarification on this is needed. The pattern I've used to generate valid agent tokens is the following:

K<69-character lower-cased alphanumerical string>::agent:<32-character lower-cased alphanumerical string>

A token generated with the format above will be 110 characters long.

[ChristianCiach]

Interesting, thanks!

I have similar questions about other settings, like --system-default-registry. The release notes say:

This setting is configured on servers, propagated to agents.

What happens if not all servers have the same value for this setting? Which value wins?

Anyway, thank you for looking into this!

[ehlesp]

On one hand, sorry but I haven't used that parameter so I cannot tell. On the other, it'll be better if you open your own question about it, so people here can find it more easily and hopefully you might get an answer on that.

[ChristianCiach]

Nah, I don't really need an answer for that. This was a completely theoretical question and I don't expect to ever find myself in such a situation (because, of course, we deploy K3s using automation, like Ansible playbooks).

The purpose of my comment was to show that your question does not necessarily only apply to the agent-token parameter, but that there are numerous other parameters where the behavior of K3s is undefined if these parameters differ between the nodes.

[ehlesp]

Got your point. That's the usual problem with any software project, documentation (when there's such a thing) tends to lag behind the current code. Not surprising, since the same people that are coding are usually the ones who have to update the docs, and only if they find the time...

[brandond]

The long token format is K10<sha256 sum of cluster CA certificate>::<username>:<password>. If you're specifying the token or agent-token on the command line of a server in order to set it, you should specify just the password portion, as the CA and username portion are set internally. The long token is only used on nodes joining the cluster, as the presence of the CA hash in the token allows them to verify the identity of the cluster they're joining. Our docs on this subject could definitely use some work.

Synchronizing server configuration across nodes is left as an exercise for the reader. If you have multiple servers, they should all have the same configuration values (except for the --cluster-init/--server flags that control cluster creation and joining). Most folks handle this via deployment automation - ansible, puppet, vagrant, etc. Failure to keep these items in sync can lead to undefined behavior.

[ehlesp]

Thanks for your clarification! Now I get it. So, if I've understood you right, the agent token has to be defined in the server nodes as follows.

server --agent-token <password>

And, in the agent nodes, the token has to be specified as below.

agent --server "https:\\10.0.0.1:6443" --token K10<sha256 sum of cluster CA certificate>::<username>:<password>

To tell you the truth, I'm wondering how all my agent nodes where able to join my cluster nevertheless by using a badly generated token. Still, my problem now is that I don't know which is the CA certificate you mention, it's the /var/lib/rancher/k3s/server/tls/client-ca.crt file present in any of the server nodes? And where I can find that <username> for agents set internally?

[brandond]

The hash can be calculated locally by running sha256sum /var/lib/rancher/k3s/server/tls/server-ca.crt or curl -ks https://localhost:6443/cacerts | sha256sum.

Agents join with a username of 'node', servers join with a username of 'server'. This is actually handled internally, IIRC it shouldn't really matter what username you specify in the long token since it is overridden.

Normally you don't have to calculate this yourself, you can just cat /var/lib/rancher/k3s/server/token

[ehlesp]

Good, understood. I'll do some testing when I can with all these things you've told me and, after that, I'll leave here my impressions.

[ehlesp]

I've created a new cluster with the agent tokens generated and set properly, and I've compared it with another cluster with bad agent tokens.

1. In both cases, the agent nodes join the cluster and reach the Ready state.
2. In the log of the agent nodes of the cluster with bad agent tokens, I've seen the following lines.

time="2021-06-16T12:24:46.046546700+02:00" level=warning msg="Cluster CA certificate is not trusted by the host CA bundle, but the token does not include a CA hash. Use the
 full token from the server's node-token file to enable Cluster CA validation."
time="2021-06-16T12:24:46.211325769+02:00" level=error msg="Failed to configure agent: https://127.0.0.1:6444/v1-k3s/serving-kubelet.crt: 503 Service Unavailable"

These lines don't appear in the log of the agent nodes of the cluster with the proper agent tokens, so I understand they are related to the issue of the agent token being incorrect.

So, the agent token seems to work as you've explained @brandond , although I find strange that agent nodes can join a cluster with badly generated and set tokens. Of course, I understand that this is still an experimental setting. Also, I think the name agent-token is kind of misleading, maybe it would be better to call it something like agent-token-pw or agent-password. This is just an opinion, I don't know what's the criteria used to name the parameters for the K3s command.