Bug 1982726 - kube-apiserver audit logs show a lot of 404 errors for DELETE "*/secrets/encryption-config" on single node clusters
Summary: kube-apiserver audit logs show a lot of 404 errors for DELETE "*/secrets/encr...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-apiserver
Version: 4.9
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: 4.10.0
Assignee: Damien Grisonnet
QA Contact: Ke Wang
URL:
Whiteboard: LifecycleReset
Depends On:
Blocks: 2029504
TreeView+ depends on / blocked
 
Reported: 2021-07-15 14:44 UTC by Omer Tuchfeld
Modified: 2022-03-12 04:36 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 2029504 (view as bug list)
Environment:
Last Closed: 2022-03-12 04:36:01 UTC
Target Upstream Version:
Embargoed:
dgrisonn: needinfo-


Attachments (Terms of Use)
relevant events, extracted with: sudo cat *audit-*.log | jq 'select(.responseStatus.code >= 400) | select(.requestURI | test("encryption"))' > enc && gzip enc (241.34 KB, application/gzip)
2021-07-15 14:44 UTC, Omer Tuchfeld
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-authentication-operator pull 506 0 None open Bug 1982726: encryption-controller: sync secrets conditionally 2021-11-16 15:22:08 UTC
Github openshift cluster-kube-apiserver-operator pull 1255 0 None open Bug 1982726: encryption-controller: sync secrets conditionally 2021-11-16 15:16:49 UTC
Github openshift cluster-openshift-apiserver-operator pull 483 0 None open Bug 1982726: encryption-controller: sync secrets conditionally 2021-11-16 15:15:19 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-12 04:36:22 UTC

Description Omer Tuchfeld 2021-07-15 14:44:03 UTC
Created attachment 1801898 [details]
relevant events, extracted with:  sudo cat *audit-*.log | jq 'select(.responseStatus.code >= 400) | select(.requestURI | test("encryption"))' > enc && gzip enc

Description of problem:
kube-apiserver audit logs show a lot of 404 errors for DELETE on "*/secrets/encryption-config" endpoints on single node clusters. This may also happen on multi-node clusters, haven't checked.

Version-Release number of selected component (if applicable)+
How reproducible+
Steps to Reproduce:

Observed during this nightly run
https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.9-e2e-aws-single-node/1415462796505649152

Actual results:
A lot of 404 errors

Expected results:
Less 404 errors

Additional info:
Attached relevant events, extracted with:

sudo cat *audit-*.log | jq 'select(.responseStatus.code >= 400) | select(.requestURI | test("encryption"))' > enc && gzip enc

Comment 1 Michal Fojtik 2021-08-14 14:53:20 UTC
This bug hasn't had any activity in the last 30 days. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. As such, we're marking this bug as "LifecycleStale" and decreasing the severity/priority. If you have further information on the current state of the bug, please update it, otherwise this bug can be closed in about 7 days. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant. Additionally, you can add LifecycleFrozen into Keywords if you think this bug should never be marked as stale. Please consult with bug assignee before you do that.

Comment 2 Lukasz Szaszkiewicz 2021-09-03 13:45:50 UTC
Iā€™m adding UpcomingSprint, because I was occupied by fixing bugs with higher priority/severity, developing new features with higher priority, or developing new features to improve stability at a macro level. I will revisit this bug next sprint.

Comment 4 Michal Fojtik 2021-11-24 15:09:38 UTC
The LifecycleStale keyword was removed because the bug moved to QE.
The bug assignee was notified.

Comment 7 Ke Wang 2021-12-02 03:12:58 UTC
Verification steps,

1. Downloaded the attachment and searched the 404 error,
$ gunzip -d enc.gz 
$ grep -c '"code": 404' enc
4894

2. Installed one single node and searched the 404 error in kube-apiserver audit log files,

$ oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.0-0.nightly-2021-12-01-164437   True        False         56m     Cluster version is 4.10.0-0.nightly-2021-12-01-164437

$ oc get no
NAME                                 STATUS   ROLES           AGE   VERSION
ci-ln-thj43g2-72292-8fghz-master-0   Ready    master,worker   77m   v1.22.1+bac83a5

$ oc debug node/<master>
sh-4.4# cd /var/log/kube-apiserver
sh-4.4# sudo cat *audit-*.log | jq 'select(.responseStatus.code >= 400) | select(.requestURI | test("encryption"))' > enc

sh-4.4# grep '"code": 404' enc | wc -l
11

Got less 404 errors than before, based this result, move the bug VERIFIED.

Comment 10 errata-xmlrpc 2022-03-12 04:36:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056


Note You need to log in before you can comment on or make changes to this bug.