Created attachment 1646796 [details] apiserver log Description of problem: cluster installed with installer version 4.2.12-s390x rhcos version 4.2.10-s390x every ~5 minutes the kube-apiserver throw the event Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert" not found) it does automatically recover but something seems to be broken here
logfile attached does not contain the error as described - is this the correct log file?
Created attachment 1649409 [details] cert syncer log
Happy new year! I see the same message on OCP V4.12 X86_64. Message is in cert-syncer container. My system is on Red Hat VPN, please ping me on Hangouts if you would like to see yourself. Cheers Dan
If the same thing is happening on x86, then that consistency would suggest its not S/390 specific - I will dig a little deeper and see whats happening. Before the error message on each line is "type: 'Warning'" so it may be just that, or the error is getting incorrectly wrapped as a warning for the event. Will look into it further.
Thx for looking into this. What makes my cluster a little bit special is that I use a Non-Public-CA as a signer for the ingress router server cert. I.e. *.apps.ocp4... is served with a cert which is signed by a CA which is per default NOT in the the RHCOS trust store (actually, it is signed by the Red Hat Internal CA). I followed the installation guidance to add the CA Trust Chain as described here: https://docs.openshift.com/container-platform/4.2/networking/configuring-a-custom-pki.html However, this is NOT true for api.ocp4..., that is using certs from openshift internal. Just wanted to mention this, as the message is about certs.
This appears to be underway already in the linked bug. That PR is for 4.3 and there is an open/pending discussion about back porting to 4.2.x
Thx. I dont need a backport for this. 4.3 is (hopefully) coming soon, and this seems not to have any impact besides being annoying. Thx again! Dan
Hi, is this PR available for 4.2.x?
I have this issue in OCP 4.2.16.
I am not aware of a fix or this in 4.2.x. According to the linked bug above, it appears to be corrected for 4.3.
Hi Carvel, There are some customers in our region who cant upgrade their cluster to 4.3 due to application compatibility issue. Can we request for this bug fix to be back-ported to v 4.2.x?
This bug racks the issue for non-x86 architectures. I think you'll need to open a new bug for the backport for x86 and reference the bug with the actual fix.
@Ashish Prajapati: Have you opened the bug for 4.2.z on x86? Let us know in order to not duplicate the issue.
*** Bug 1806089 has been marked as a duplicate of this bug. ***
(Just fyi) per above comments (TL;DR), not sure if this bug is duplicated with bug 1780243#c2 .
Verified with OCP build 4.2.0-0.nightly-2020-03-16-141929, $ oc get events |grep "Removed file for secret" Nothing found. So move the bug verify.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0936