contextual logging #3077

pohly · 2021-12-06T10:32:26Z

Enhancement Description

Current configuration

https://github.com/kubernetes/kubernetes/blob/master/hack/logcheck.conf

Status

The following table counts log calls that need to be converted. The numbers for contextual logging include those for structured logging.

At this point, controllers could get converted to contextual logging or one of the components that was already converted to structured logging. If you want to pick one, ping @pohly on the #wg-structured-logging Slack channel. See structured and contextual logging migration instructions for guidance.

Besides migrating log calls, we also might have to migrate from APIs which don't support contextual logging to APIs which do:

k8s.io/client-go/tools/[events|record]: add WithContext
- cronjob controller: reconsider log verbosity kubernetes#115574
HandlePanic/HandleError -> HandlePanicWithContext, HandleErrorWithContext

From 2022-10-27 ~= Kubernetes 1.26

The focus was on converting kube-controller-manager. Of 1944 unstructured and/or non-contexual logging calls in pkg/controller and cmd/kube-controller-manager, 82% were converted to structured, contextual logging in Kubernetes 1.27.

Component	Non-Structured Logging	Non-Contextual Logging	Owner
pkg/controller/bootstrap	15	28	@mengjiao-liu, kubernetes/kubernetes#113464
pkg/controller/certificates	22	31	@mengjiao-liu, kubernetes/kubernetes#113994
pkg/controller/clusterroleaggregation	2	2	@mengjiao-liu, kubernetes/kubernetes#113910
pkg/controller/cronjob	1	44	@mengjiao-liu, kubernetes/kubernetes#113428
pkg/controller/daemon	45	85	@249043822, kubernetes/kubernetes#113622
pkg/controller/deployment	23	79	@249043822, kubernetes/kubernetes#113525
pkg/controller/disruption	29	56	@Namanl2001, kubernetes/kubernetes#116021
pkg/controller/endpoint	12	24	lunhuijie (Slack)
pkg/controller/endpointslice	22	36	@Namanl2001, kubernetes/kubernetes#115295
pkg/controller/endpointslicemirroring	18	28	@Namanl2001, kubernetes/kubernetes#114982
pkg/controller/garbagecollector	55	105	@ncdc, kubernetes/kubernetes#113471
pkg/controller/job	12	36	was: @sanwishe, kubernetes/kubernetes#113576, now: @mengjiao-liu
pkg/controller/namespace	30	55	@yangjunmyfm192085, kubernetes/kubernetes#113443
pkg/controller/nodeipam	135	210	@yangjunmyfm192085, kubernetes/kubernetes#112670
pkg/controller/nodelifecycle	60	106	@yangjunmyfm192085, kubernetes/kubernetes#112670
pkg/controller/podautoscaler	9	13	@freddie400, kubernetes/kubernetes#114687
pkg/controller/podgc	10	24	@pravarag, kubernetes/kubernetes#114689
pkg/controller/replicaset	20	49	@Namanl2001, kubernetes/kubernetes#114871
pkg/controller/resourcequota	24	37	@ncdc, kubernetes/kubernetes#113315
pkg/controller/serviceaccount	22	31	@Namanl2001, kubernetes/kubernetes#114918
pkg/controller/statefulset	19	59	@249043822, kubernetes/kubernetes#113840
pkg/controller/storageversiongc	4	6	@songxiao-wang87, kubernetes/kubernetes#113986
pkg/controller/testutil	9	9	@Octopusjust, kubernetes/kubernetes#114061
pkg/controller/ttl	4	8	wxs (Slack) = @songxiao-wang87, kubernetes/kubernetes#113916
pkg/controller/ttlafterfinished	9	15	@obaranov1, kubernetes/kubernetes#115332
pkg/controller/util	0	19	@fatsheep9146, kubernetes/kubernetes#115049
pkg/controller/volume	351	673	@yangjunmyfm192085, kubernetes/kubernetes#113584
pkg/kubelet	1	1805	@fmuyassarov
pkg/scheduler	0	348	@knelasevero, kubernetes/kubernetes#111155
staging/src/k8s.io/apiextensions-apiserver	57	81
staging/src/k8s.io/apimachinery	73	114	@yanjing1104
staging/src/k8s.io/apiserver	262	543
staging/src/k8s.io/client-go	161	267
staging/src/k8s.io/cloud-provider	108	146
staging/src/k8s.io/cluster-bootstrap	2	4
staging/src/k8s.io/code-generator	108	168
staging/src/k8s.io/component-base	32	63
staging/src/k8s.io/component-helpers	7	8
staging/src/k8s.io/controller-manager	10	10
staging/src/k8s.io/csi-translation-lib	3	4
staging/src/k8s.io/kube-aggregator	52	76
staging/src/k8s.io/kube-controller-manager	0	0
staging/src/k8s.io/kubectl	89	147	@yanjing1104
staging/src/k8s.io/legacy-cloud-providers	1445	2238
staging/src/k8s.io/mount-utils	54	92
staging/src/k8s.io/pod-security-admission	1	34	@Namanl2001, kubernetes/kubernetes#114471
staging/src/k8s.io/sample-controller	16	22	@pchan, kubernetes/kubernetes#113879

From 2023-03-17 = Kubernetes v1.27.0-beta.0

All of kube-controller-manager got converted.

Tables created with:

go install sigs.k8s.io/logtools/logcheck@latest

echo "Component | Non-Structured Logging | Non-Contextual Logging | Owner " && \
echo "------ | ------- | ------" && \
for i in $(find pkg/controller/* pkg/scheduler pkg/kubelet pkg/apis pkg/api cmd/kube-* cmd/kubelet  -maxdepth 0 -type d | sort); do \
     echo "$i | $(cd $i; ${GOPATH}/bin/logcheck -check-structured -check-deprecations=false 2>&1 ./... | wc -l ) | $(cd $i; ${GOPATH}/bin/logcheck -check-structured -check-deprecations=false -check-contextual ./... 2>&1 | wc -l ) |" | grep -v '| 0 | 0 |'; \
done

Component	Non-Structured Logging	Non-Contextual Logging	Owner
cmd/kube-apiserver	7	8	on hold
cmd/kubelet	0	47	@fmuyassarov
cmd/kube-proxy	0	46	on hold
pkg/controller/certificates	22	31	@mengjiao-liu, kubernetes/kubernetes#113994
pkg/controller/deployment	2	5	@fatsheep9146, kubernetes/kubernetes#116930
pkg/controller/disruption	29	54	~~@obaranov1, kubernetes/kubernetes#116021~~, @mengjiao-liu, kubernetes/kubernetes#119147
pkg/controller/endpoint	12	24	@my-git9, kubernetes/kubernetes#116755
pkg/controller/endpointslice	20	35	@Namanl2001, kubernetes/kubernetes#115295
pkg/controller/endpointslicemirroring	18	28	@Namanl2001, kubernetes/kubernetes#114982
pkg/controller/garbagecollector	3	3	@fatsheep9146, kubernetes/kubernetes#116930
pkg/controller/job	12	35	@sanwishe, kubernetes/kubernetes#113576 (needs new owner?)
pkg/controller/nodeipam	8	13	@fatsheep9146, kubernetes/kubernetes#116930
pkg/controller/podgc	10	24	~~@pravarag, kubernetes/kubernetes#114689~~, @pohly, kubernetes/kubernetes#119250
pkg/controller/replicaset	9	18	@fatsheep9146, kubernetes/kubernetes#116930
pkg/controller/statefulset	3	5	@kerthcet, kubernetes/kubernetes#118071
pkg/controller/testutil	9	9	@Octopusjust, kubernetes/kubernetes#114061
pkg/controller/util	0	4	@fatsheep9146, kubernetes/kubernetes#116930
pkg/controller/volume	5	20	@fatsheep9146, kubernetes/kubernetes#116930
pkg/kubelet	2	1923	@fmuyassarov, kubernetes/kubernetes#114352
pkg/scheduler	2	349	@mengjiao-liu, kubernetes/kubernetes#91633

From 2023-09-18 =~ Kubernetes v1.28

Component	Non-Structured Logging	Non-Contextual Logging	Owner
cmd/kube-apiserver	6	7	on hold
cmd/kubelet	0	52	@fmuyassarov (?), kubernetes/kubernetes#114352
cmd/kube-proxy	0	41	on hold
pkg/kubelet	2	1942	@fmuyassarov (?)
pkg/scheduler	1	137	@mengjiao-liu, https://github.com/kubernetes/kubernetes/pulls/mengjiao-liu
staging/src/k8s.io/apiserver	?	?	@tallclair, kubernetes/kubernetes#114198
staging/src/k8s.io/client-go/discovery	11	21	on hold
staging/src/k8s.io/client-go/examples	14	14	on hold
staging/src/k8s.io/client-go/metadata	2	4	on hold
staging/src/k8s.io/client-go/plugin	5	8	on hold
staging/src/k8s.io/client-go/rest	16	37	on hold
staging/src/k8s.io/client-go/restmapper	3	6	on hold
staging/src/k8s.io/client-go/tools	104	171	@pohly, kubernetes/kubernetes#120729
staging/src/k8s.io/client-go/transport	17	31	on hold
staging/src/k8s.io/client-go/util	12	19	on hold

Table created manually and with:

go install sigs.k8s.io/logtools/logcheck@latest

echo "Component | Non-Structured Logging | Non-Contextual Logging | Owner " && \
echo "------ | ------- |  ------- | ------" && \
for i in $(find pkg/scheduler pkg/kubelet pkg/apis pkg/api cmd/kube-* cmd/kubelet staging/src/k8s.io/client-go/* -maxdepth 0 -type d | sort); do \
     echo "$i | $(cd $i; ${GOPATH}/bin/logcheck -check-structured -check-deprecations=false 2>&1 ./... | wc -l ) | $(cd $i; ${GOPATH}/bin/logcheck -check-structured -check-deprecations=false -check-contextual ./... 2>&1 | wc -l ) |" | grep -v '| 0 | 0 |'; \
done

From 2023-11-20 =~ Kubernetes v1.29

Component	Non-Structured Logging	Non-Contextual Logging	Owner
cmd/kube-apiserver	6	7	@tallclair
cmd/kubelet	0	52	@fmuyassarov (?), kubernetes/kubernetes#114352
pkg/kubelet	2	1983	@fmuyassarov
cmd/kube-proxy	0	42	@ fatsheep9146, kubernetes/kubernetes#122197
pkg/proxy	0	360	@ fatsheep9146, see above
staging/src/k8s.io/apiserver	285	655	@tallclair, kubernetes/kubernetes#114198
staging/src/k8s.io/client-go/discovery	11	21
staging/src/k8s.io/client-go/examples	14	14
staging/src/k8s.io/client-go/metadata	2	4
staging/src/k8s.io/client-go/plugin	5	8
staging/src/k8s.io/client-go/rest	16	37
staging/src/k8s.io/client-go/restmapper	3	6
staging/src/k8s.io/client-go/tools	83	143	@pohly
staging/src/k8s.io/client-go/transport	17	31
staging/src/k8s.io/client-go/util	12	19

Table created with:

go install sigs.k8s.io/logtools/logcheck@latest

echo "Component | Non-Structured Logging | Non-Contextual Logging | Owner " && echo "------ | ------- |  ------- | ------" && for i in $(find pkg/scheduler pkg/kubelet pkg/apis pkg/api cmd/kube-* cmd/kubelet staging/src/k8s.io/client-go/* staging/src/k8s.io/apiserver -maxdepth 0 -type d | sort); do      echo "$i | $(cd $i; ${GOPATH}/bin/logcheck -check-structured -check-deprecations=false 2>&1 ./... | wc -l ) | $(cd $i; ${GOPATH}/bin/logcheck -check-structured -check-deprecations=false -check-contextual ./... 2>&1 | wc -l ) |" | grep -v '| 0 | 0 |'; done

The text was updated successfully, but these errors were encountered:

pohly · 2021-12-08T12:37:28Z

/sig instrumentation
/wg structured-logging

hosseinsalahi · 2022-01-21T10:03:34Z

Hello @pohly

v1.24 Enhancements team here.

Just checking in as we approach enhancements freeze on 18:00pm PT on Thursday Feb 3rd, 2022. This enhancement is targeting alpha for v1.24,

Here’s where this enhancement currently stands:

Updated KEP file using the latest template has been merged into the k/enhancements repo.
KEP status is marked as implementable for this release
KEP has a test plan section filled out.
KEP has up to date graduation criteria.
KEP has a production readiness review that has been completed and merged into k/enhancements.

The status of this enhancement is marked as tracked. Please keep the issue description and the targeted stage up-to-date for release v1.24.
Thanks!

pohly · 2022-02-03T06:44:26Z

@encodeflush : the KEP PR was merged, all criteria for alpha in 1.24 should be met now.

chrisnegus · 2022-02-11T18:51:41Z

Hi @pohly 👋 1.24 Docs shadow here.

This enhancement is marked as 'Needs Docs' for the 1.24 release.

Please follow the steps detailed in the documentation to open a PR against the dev-1.24 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thu March 31, 11:59 PM PDT.

Also, if needed take a look at Documenting for a release to familiarize yourself with the docs requirement for the release.

Thanks!

valaparthvi · 2022-03-21T04:13:22Z

Hi @pohly 👋 1.24 Release Comms team here.

We have an opt-in process for the feature blog delivery. If you would like to publish a feature blog for this issue in this cycle, then please opt in on this tracking sheet.

The deadline for submissions and the feature blog freeze is scheduled for 01:00 UTC Wednesday 23rd March 2022 / 18:00 PDT Tuesday 22nd March 2022. Other important dates for delivery and review are listed here: https://github.com/kubernetes/sig-release/tree/master/releases/release-1.24#timeline.

For reference, here is the blog for 1.23.

Please feel free to reach out any time to me or on the #release-comms channel with questions or comments.

Thanks!

hosseinsalahi · 2022-03-21T17:13:46Z

Hello @pohly

I'm just checking in once more as we approach the 1.24 Code Freeze on 18:00 PDT, Tuesday, March 29th 2022

Please ensure the following items are completed:

All PRs to the Kubernetes repo that are related to your enhancement are merged by the code freeze deadline.
Have a documentation placeholder PR open by 18:00 PDT, Thursday, March 31, 2022.
- features: add ContextualLogging website#32452
- logging: include information about contextual logging community#6560

For note, the status of this enhancement is currently marked as tracked.

Thank you!

pohly · 2022-03-23T16:16:49Z

/assign

pohly · 2022-03-23T16:48:29Z

I have added two doc PRs to the description.

Priyankasaggu11929 · 2022-05-10T04:54:32Z

/milestone clear

logicalhan · 2022-05-12T16:38:58Z

@pohly can we close this?

logicalhan · 2022-05-12T16:40:00Z

/assign @serathius

pohly · 2024-02-22T14:18:56Z

Doc PR for 1.30 created and linked to in the description

tjons · 2024-02-25T18:38:21Z

Hey again @pohly 👋 Enhancements team here,

Just checking in as we approach code freeze at 02:00 UTC Wednesday 6th March 2024 .

Here's where this enhancement currently stands:

All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
All PR/s are ready to be merged (they have approved and lgtm labels applied) by the code freeze deadline. This includes tests.

For this enhancement, it looks like the following PRs are open and need to be merged before code freeze:

Also, please let me know if there are other PRs in k/k we should be tracking for this KEP.
As always, we are here to help if any questions come up. Thanks!

tjons · 2024-03-06T02:23:51Z

Hi @pohly - checking in again here. There are about ~6 hours remaining until code freeze. Do you think you'll be able to merge kubernetes/kubernetes#120696 in time?

salehsedghpour · 2024-03-06T08:59:59Z

Hello @pohly 👋 , Enhancements team here.

With all the implementation(code related) PRs merged as per the issue description:

promote contextual logging to beta, enabled by default kubernetes#122589

This enhancement is now marked as tracked for code freeze for the v1.30 Code Freeze!

sreeram-venkitesh · 2024-05-14T18:13:48Z

Hi @pohly 👋, 1.31 Enhancements Lead here.

If you wish to progress this enhancement in v1.31, please opt-in your enhancement by adding the lead-opted-in label and set the milestone to v1.31 before the Production Readiness Review Freeze.

/remove-label lead-opted-in

k8s-triage-robot · 2024-08-12T18:27:53Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

ffromani · 2024-08-13T09:34:23Z

/remove-lifecycle stale

k8s-triage-robot · 2024-11-11T09:41:00Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

ffromani · 2024-11-11T10:27:18Z

/remove-lifecycle stale

k8s-ci-robot added the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Dec 6, 2021

pohly mentioned this issue Dec 6, 2021

KEP-3077: contextual logging #3078

Merged

k8s-ci-robot added sig/instrumentation Categorizes an issue or PR as relevant to SIG Instrumentation. wg/structured-logging Categorizes an issue or PR as relevant to WG Structured Logging. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Dec 8, 2021

ehashman mentioned this issue Jan 14, 2022

New unique logging meta-data into key Kubernetes logs #1668

Closed

gracenng added the tracked/yes Denotes an enhancement issue is actively being tracked by the Release Team label Jan 17, 2022

gracenng added this to the v1.24 milestone Jan 17, 2022

This was referenced Feb 14, 2022

KEP 3077: contextual logging: simplify scope, avoid transition #3222

Merged

contextual logging kubernetes/klog#296

Merged

logcheck: contextual logging + enhanced checks kubernetes/klog#297

Merged

k8s-ci-robot assigned pohly Mar 23, 2022

This was referenced Mar 24, 2022

contextual logging kubernetes/kubernetes#108995

Merged

contextual logging blog post kubernetes/website#32656

Merged

mehabhalodiya mentioned this issue Apr 4, 2022

features: add ContextualLogging kubernetes/website#32452

Merged

github-actions bot mentioned this issue May 6, 2022

Blog: Kubernetes 1.24: Stargazer rajula96reddy/rss-issue-action-test#18

Open

k8s-ci-robot removed this from the v1.24 milestone May 10, 2022

Priyankasaggu11929 added tracked/no Denotes an enhancement issue is NOT actively being tracked by the Release Team and removed tracked/yes Denotes an enhancement issue is actively being tracked by the Release Team labels May 10, 2022

tjons moved this from Tracked for Enhancements Freeze to At Risk for Code Freeze in 1.30 Enhancements Tracking Feb 25, 2024

salehsedghpour moved this from At Risk for Code Freeze to Tracked for Code Freeze in 1.30 Enhancements Tracking Mar 6, 2024

drewhagen moved this from Tracked for Code Freeze to Tracked for Doc Freeze in 1.30 Enhancements Tracking Apr 1, 2024

shafeeqes mentioned this issue Apr 30, 2024

☂️-Issue for "Support for Kubernetes v1.30” gardener/gardener#9508

Closed

16 tasks

mengjiao-liu mentioned this issue May 8, 2024

2023 Annual Report: WG Structured Logging kubernetes/community#7813

Merged

k8s-infra-ci-robot added this to 1.31 Enhancements Tracking May 13, 2024

k8s-ci-robot removed the lead-opted-in Denotes that an issue has been opted in to a release label May 14, 2024

sreeram-venkitesh removed this from 1.31 Enhancements Tracking May 14, 2024

zwpaper mentioned this issue May 18, 2024

Migrate to contextual logging kubernetes-sigs/scheduler-plugins#743

Closed

4 tasks

ricardoapl mentioned this issue May 28, 2024

[WIP] Migrate client-go/discovery to structured logging kubernetes/kubernetes#125161

Closed

liyuerich mentioned this issue Jul 10, 2024

migrate pkg/server/dynamiccertificates to contextual logging kubernetes/kubernetes#126002

Open

pohly mentioned this issue Jul 26, 2024

add alternative APIs which support contextual logging kubernetes/kubernetes#126379

Open

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 12, 2024

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 13, 2024

pacoxu mentioned this issue Oct 14, 2024

feat(import-boss): support structured and contextual logging kubernetes/kubernetes#128024

Closed

carlory mentioned this issue Oct 24, 2024

kcm should use the controller name constants kubernetes/kubernetes#127550

Open

yanmxa mentioned this issue Oct 29, 2024

✨ Switch zap log for global hub agent stolostron/multicluster-global-hub#1184

Merged

4 tasks

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Nov 11, 2024

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Nov 11, 2024

serathius removed their assignment Nov 11, 2024

pohly mentioned this issue Nov 21, 2024

kubelet not using full timestamp precision in its log records kubernetes/klog#414

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

contextual logging #3077

contextual logging #3077

pohly commented Dec 6, 2021 •

edited

Loading

pohly commented Dec 8, 2021

hosseinsalahi commented Jan 21, 2022 •

edited

Loading

pohly commented Feb 3, 2022

chrisnegus commented Feb 11, 2022

valaparthvi commented Mar 21, 2022 •

edited

Loading

hosseinsalahi commented Mar 21, 2022 •

edited

Loading

pohly commented Mar 23, 2022

pohly commented Mar 23, 2022

Priyankasaggu11929 commented May 10, 2022

logicalhan commented May 12, 2022

logicalhan commented May 12, 2022

pohly commented Feb 22, 2024

tjons commented Feb 25, 2024

tjons commented Mar 6, 2024

salehsedghpour commented Mar 6, 2024

sreeram-venkitesh commented May 14, 2024

k8s-triage-robot commented Aug 12, 2024

ffromani commented Aug 13, 2024

k8s-triage-robot commented Nov 11, 2024

ffromani commented Nov 11, 2024

contextual logging #3077

contextual logging #3077

Comments

pohly commented Dec 6, 2021 • edited Loading

Enhancement Description

Current configuration

Status

From 2022-10-27 ~= Kubernetes 1.26

From 2023-03-17 = Kubernetes v1.27.0-beta.0

From 2023-09-18 =~ Kubernetes v1.28

From 2023-11-20 =~ Kubernetes v1.29

pohly commented Dec 8, 2021

hosseinsalahi commented Jan 21, 2022 • edited Loading

pohly commented Feb 3, 2022

chrisnegus commented Feb 11, 2022

valaparthvi commented Mar 21, 2022 • edited Loading

hosseinsalahi commented Mar 21, 2022 • edited Loading

pohly commented Mar 23, 2022

pohly commented Mar 23, 2022

Priyankasaggu11929 commented May 10, 2022

logicalhan commented May 12, 2022

logicalhan commented May 12, 2022

pohly commented Feb 22, 2024

tjons commented Feb 25, 2024

tjons commented Mar 6, 2024

salehsedghpour commented Mar 6, 2024

sreeram-venkitesh commented May 14, 2024

k8s-triage-robot commented Aug 12, 2024

ffromani commented Aug 13, 2024

k8s-triage-robot commented Nov 11, 2024

ffromani commented Nov 11, 2024

pohly commented Dec 6, 2021 •

edited

Loading

hosseinsalahi commented Jan 21, 2022 •

edited

Loading

valaparthvi commented Mar 21, 2022 •

edited

Loading

hosseinsalahi commented Mar 21, 2022 •

edited

Loading