[sig-arch][Late] clients should not use APIs that are removed in upcoming releases [Suite:openshift/conformance/parallel] Is flaking frequently in CI, and also indicates that the component is using APIs that are due to be removed in 2 releases. See: https://sippy.ci.openshift.org/sippy-ng/tests/4.10/analysis?test=%5Bsig-arch%5D%5BLate%5D%20clients%20should%20not%20use%20APIs%20that%20are%20removed%20in%20upcoming%20releases%20%5BSuite%3Aopenshift%2Fconformance%2Fparallel%5D A job demonstrating the problem: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.10-e2e-aws-workers-rhel8/1445003313710895104 Prow output: user/system:serviceaccount:openshift-monitoring:kube-state-metrics accessed cronjobs.v1beta1.batch 12 times user/system:serviceaccount:openshift-monitoring:kube-state-metrics accessed poddisruptionbudgets.v1beta1.policy 11 times This problem is showing significantly more in 4.10 than 4.9: https://search.ci.openshift.org/?search=user%2Fsystem%3Aserviceaccount%3Aopenshift-monitoring%3Akube-state-metrics+accessed&maxAge=48h&context=1&type=bug%2Bjunit&name=4.10&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job 4.10 shows this appears in 39% of ci runs over the past 2 days. 4.9 shows it at 7%.
Leaving it to Damien for appreciation. As for why the test fails more often in 4.10 than 4.9, I can only think that the test itself is based on more accurate data. From kube-state-metrics standpoint, nothing has changed significantly between the 2 releases (e.g. both use the same API groups). For the record: https://kubernetes.io/docs/reference/using-api/deprecation-guide/#poddisruptionbudget-v125 https://kubernetes.io/docs/reference/using-api/deprecation-guide/#cronjob-v125
Assigning to Filip for clarification. My understanding is that kube-state-metrics would stop using the deprecated APIs once it decides to support Kube v1.25 (which would also mean "stop supporting v1.20"). Looking at the compatibility matrix [1], I'm not sure about the exact timing though. [1] https://github.com/kubernetes/kube-state-metrics#compatibility-matrix
I will bring this topic up in upstream and try to get it resolved soon.
Since KSM provides support for the current + past 2 Kubernetes versions, it will drop support for these APIs and switch to stable ones only when Kubernetes 1.23 is released. Is this a problem for us?
*** Bug 2029743 has been marked as a duplicate of this bug. ***
Fixed by https://github.com/openshift/kube-state-metrics/pull/67 which bumps kube-state-metrics to v2.4.2 (including https://github.com/kubernetes/kube-state-metrics/commit/d01ad03ae64661f37caa79b97ad8a4edf2ac3933).
we only fix the error in openshift-monitoring, for other errors, should be fixed by the related team, example: ******************************************* user/system:serviceaccount:openshift-insights:gather accessed podsecuritypolicies.v1beta1.policy 1 times user/system:serviceaccount:openshift-monitoring:kube-state-metrics accessed cronjobs.v1beta1.batch 9 times user/system:serviceaccount:openshift-monitoring:kube-state-metrics accessed horizontalpodautoscalers.v2beta2.autoscaling 29 times user/system:serviceaccount:openshift-monitoring:kube-state-metrics accessed poddisruptionbudgets.v1beta1.policy 9 times user/system:serviceaccount:openshift-network-operator:default accessed flowschemas.v1beta1.flowcontrol.apiserver.k8s.io 127 times ******************************************* podsecuritypolicies is owned by openshift-insights team, see Bug 2073945 checked with 4.11.0-0.nightly-2022-04-16-163450, # oc -n openshift-monitoring logs -c kube-state-metrics $(oc -n openshift-monitoring get pod --no-headers | grep kube-state-metrics | awk '{print $1}') I0418 23:25:03.772386 1 server.go:93] Using default resources I0418 23:25:03.772500 1 types.go:136] Using all namespace I0418 23:25:03.773814 1 server.go:122] Metric allow-denylisting: Excluding the following lists that were on denylist: ^kube_pod_container_status_running$, ^kube_pod_completion_time$, ^kube_.+_created$, ^kube_.+_annotations$, ^kube_.+_metadata_resource_version$, ^kube_pod_restart_policy$, ^kube_replicaset_status_observed_generation$, ^kube_pod_init_container_status_terminated$, ^kube_pod_container_status_terminated$, ^kube_pod_status_scheduled$, ^kube_pod_init_container_status_running$, ^kube_secret_labels$, ^kube_replicaset_metadata_generation$ W0418 23:25:03.773845 1 client_config.go:617] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. I0418 23:25:03.782957 1 server.go:250] Testing communication with server I0418 23:25:06.877986 1 server.go:255] Running with Kubernetes cluster version: v1.23. git version: v1.23.3+e7218b0. git tree state: clean. commit: 6f5a5295923a614a4202a7ad274b38b69f9ca8c0. platform: linux/amd64 I0418 23:25:06.878004 1 server.go:257] Communication with server successful I0418 23:25:06.878257 1 server.go:202] Starting metrics server: 127.0.0.1:8081 I0418 23:25:06.878276 1 server.go:191] Starting kube-state-metrics self metrics server: 127.0.0.1:8082 I0418 23:25:06.878275 1 metrics_handler.go:96] Autosharding disabled I0418 23:25:06.878485 1 server.go:66] levelinfomsgTLS is disabled.http2false I0418 23:25:06.878520 1 server.go:66] levelinfomsgTLS is disabled.http2false I0418 23:25:06.879903 1 builder.go:232] Active resources: certificatesigningrequests,configmaps,cronjobs,daemonsets,deployments,endpoints,horizontalpodautoscalers,ingresses,jobs,leases,limitranges,mutatingwebhookconfigurations,namespaces,networkpolicies,nodes,persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas,secrets,services,statefulsets,storageclasses,validatingwebhookconfigurations,volumeattachments W0418 23:25:06.889705 1 warnings.go:70] autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler W0418 23:25:06.902385 1 warnings.go:70] autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler W0418 23:31:32.902847 1 warnings.go:70] autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler W0418 23:31:52.404261 1 warnings.go:70] autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler W0418 23:41:32.407128 1 warnings.go:70] autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler W0418 23:48:56.409424 1 warnings.go:70] autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler W0418 23:55:02.411621 1 warnings.go:70] autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler W0419 00:00:39.413974 1 warnings.go:70] autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler W0419 00:09:44.417159 1 warnings.go:70] autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler this warning is tracked in Bug 2069543, and this is the only error in kube-state-metrics now, since it is fixed in 4.11, we only need to check the 4.11 e2e jobs, for the periodic-ci-openshift-release-master-ci-4.11-upgrade-from-stable-4.10* job, since the issue is not fixed in 4.11, we would see the error in the 4.10 upgrade to 4.11 jobs
see from comment 11, the error in kube-state-metrics is fixed, move to VERIFIED
As note by Junqi, the "autoscaling/v2beta2 HorizontalPodAutoscaler" warning is the only one left from the list. It is already tracked upstream (https://github.com/kubernetes/kube-state-metrics/issues/1711) and because of their policy, it will be fixed there only once Kube v1.25 is released (see bug 2069543 too).
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:5069