Description of problem (please be detailed as possible and provide log
snippests):

I have been working with my customer and Noobaa has an issue when the internal certificates are rotated:

- Certificates rotated internally on this cluster on the 28th August
- From the Noobaa endpoint:

Doing the pre check on the noobaa certificate now.

sh-4.4$ openssl s_client -connect s3.openshift-storage.svc.cluster.local:443 -showcerts 2>/dev/null </dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p'
-----BEGIN CERTIFICATE-----
MIID1jCCAr6gAwIBAgIIHHJvZg8H90wwDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE
Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTY1OTExMTQzMDAe
Fw0yMjA5MTMxNTI5NDlaFw0yNDA5MTIxNTI5NTBaMCMxITAfBgNVBAMTGHMzLm9w
ZW5zaGlmdC1zdG9yYWdlLnN2YzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
ggEBAKyt3new40UT+bzHG39SLSm5XhcYt+WkrjItgn+cYIzXwRmZVehYS0g2VVCL
zuCqCqOYkl80FQk0VXIN+on5yuBAEhm4Iu51KvXb8LQL+Gd+jgCzxagv1ar45izq
f9YqmPpfXDHwtVQKeYt9qUxgcZJJ3u+a0hpqlw36kVRc8lNOtLlnDo6c4fJj6mZT
HIfHUIpZp05eXcHYPMiEGXUrV4IfzbJ8aMMT8E00rILsqlQITB3m3HMDox4f2Sns
asP5nhTVx5boJsbFhoD1Btc2nXxr5h4rwH2cGGRxkeJ5yWJGIo+n9CdF/olD5MBW
FIZLHZ9owXXbdza7H3zOcYchwLkCAwEAAaOB+jCB9zAOBgNVHQ8BAf8EBAMCBaAw
EwYDVR0lBAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQU60OA
IvfY0a/LLlTt9t6bNmmpZs4wHwYDVR0jBBgwFoAUHDbfLlYxj2u6FCpI2wtBNswR
KT0wSwYDVR0RBEQwQoIYczMub3BlbnNoaWZ0LXN0b3JhZ2Uuc3ZjgiZzMy5vcGVu
c2hpZnQtc3RvcmFnZS5zdmMuY2x1c3Rlci5sb2NhbDA1BgsrBgEEAZIIEWQCAQQm
EyQxYWViODlhZi1iMzdmLTRmNGYtYTllNS0xNWFhODk5MjJkMjEwDQYJKoZIhvcN
AQELBQADggEBAIvwEySqdjuwXjx+RDFLelDgtUkwtR9j20CYrWTSM0qE2qrAa1VR
0/cViY0/jmp/8xwuYl+3pvNSpntECz4MXgp+YNebXfewJnnlDQKAtYVpCJnahrfC
AFNFitqU+ZwABnbs7Awb8gjvlHbgYDC4G8UR3tUV+v2nnWvNt4gikGKwKpz7YgwW
rua3PrEdFZJF2TA/LbKFaPUZF6oManlX2b4gC7SmhAszeuQCvnY05GhluqRwrtBH
pmLUaUY8DmzIopq44CZcM8850JNL/p+Ds0MxHdoJqsYePjo4m2W2JJCebuEEcovq
diXJ62eEIQDkxRUZvz82M1sOrJ8CX7S/shY=
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIDUTCCAjmgAwIBAgIIVzTOuAujXdgwDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE
Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTY1OTExMTQzMDAe
Fw0yMjA3MjkxNjE3MDlaFw0yNDA5MjYxNjE3MTBaMDYxNDAyBgNVBAMMK29wZW5z
aGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE2NTkxMTE0MzAwggEiMA0GCSqG
SIb3DQEBAQUAA4IBDwAwggEKAoIBAQC0Xz8aawepXoSeYjhzK9Bg0yDeI1t2QnrR
+JoZQt/PKV/URwazCdHZQRiKH6k5n+M99uUxTh7Uw4qNRoX6xzp5xddYspmDaKtp
8YKDPWH2VJ9GKDLqCBEbH3FDZTCTgz3Vhp0iYkfCNbxN0w6eOqf3thrJ6SqSwevd
UngDAHufVJjntBmoJJ+30+htMGK79Ix9RZSxvV8nWmS1EosmAhYtcLMCTJD8VnqY
eAi5lJ8SE4XKayW1ISM+SR69DNIj+WgKFACmGx826nGkr84b2WOjkPH51bPyEFkx
jrmDltuIAzAhtByY1csZ5/lUN9A1LBXsYUA/HwDA1aG0IwWn7B1jAgMBAAGjYzBh
MA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQcNt8u
VjGPa7oUKkjbC0E2zBEpPTAfBgNVHSMEGDAWgBQcNt8uVjGPa7oUKkjbC0E2zBEp
PTANBgkqhkiG9w0BAQsFAAOCAQEAioqIEBfNBn1DGqogGjIQIZv5oc9MR3bgWOx1
6ilOBX/p0CzM6qQnaMDZYFbsF2up3oD1vlMmE/P0IEyTbryDbQsHRaDhrR4pVivB
3NkuFPPP3RbWtes9BBVuE4VnK9/gqT08U+FOOVd6h8vp6DgC8k438RNo0U12CPQF
xUkTkW+ZExR2pSi/fghGcQ3z8oZcQMsfO9W1sco6i0uyzjD0mt9UeWzHWQ/v3hSf
DVizES+B/5fB8jynBiEqoSq9CcGKnVOeCtOGN7e6nmebcP1sIg3NF5G72rf4SBFJ
WRfHjx1CseLINDhrIeRnogiMCYx2o+D8Vb++9CeglweyP6wqpw==
-----END CERTIFICATE-----

This is the old pre rotation certificate

After the noobaa end point POD restart the real live in RAM cert changes to what is recorded in the secret (tmpfs). See below when connecting to noobaa service (noobaa endpoint POD).

sh-4.4$ openssl s_client -connect s3.openshift-storage.svc.cluster.local:443 -showcerts 2>/dev/null </dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p'
-----BEGIN CERTIFICATE-----
MIID1jCCAr6gAwIBAgIIc9b+K9v7+2YwDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE
Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTY1OTExMTQzMDAe
Fw0yMzA4MjgxNjIxMTBaFw0yNTA4MjcxNjIxMTFaMCMxITAfBgNVBAMTGHMzLm9w
ZW5zaGlmdC1zdG9yYWdlLnN2YzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
ggEBAMDeoLwb1rRidBn/kkzyG8An+08flbkVJEp4FMFSFRnFhJD6C8wiKU1+FELb
cZo5L+4oS757SaP5VO7qgNDSRgaq3spyp0LJwgAxe4IjJmAHxgqFauciVVK1qUhw
R1z/5mbACNvL1bX9E+tWlDfaj7oc6oUfDqt8ni2BV/t3rZJX0lOMyq7FxLIhpotf
22ZEadf5tcWq8ZvNqsRzo6q/DwRdu5mkFhVX7SyqcTi4gyLKPwr2IvAbd1RttTLj
9DRjyEjqy2HVnv8omVEwH0wzi0AgfekG/uEmbJtcIJCMwZ+utzmyIjllgmUvM0wB
Tzd4PCLCtUVwpz9tftmuB9gPX8ECAwEAAaOB+jCB9zAOBgNVHQ8BAf8EBAMCBaAw
EwYDVR0lBAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUyJqf
MCBgzjjeN6qlsayKTQX/kIUwHwYDVR0jBBgwFoAUHlBdO0N76T6tlxhXNb/cRKj2
agAwSwYDVR0RBEQwQoIYczMub3BlbnNoaWZ0LXN0b3JhZ2Uuc3ZjgiZzMy5vcGVu
c2hpZnQtc3RvcmFnZS5zdmMuY2x1c3Rlci5sb2NhbDA1BgsrBgEEAZIIEWQCAQQm
EyQxYWViODlhZi1iMzdmLTRmNGYtYTllNS0xNWFhODk5MjJkMjEwDQYJKoZIhvcN
AQELBQADggEBABzklTqvnlw4i04V0y8OTKiVnjxuJs5zVO+EeeBmuhKb5f/O+KW9
o66WB9r4158sJpLVfVU+bGoSyhWtNGkYpHDHCFCoDqT4QdzSpVQbKi32tbACRlJe
4NFwViVUrZU0IeTjBbX6hoWBRMb6fPlEHSi9mAKYpV1PfZpTHoDHZTFhEFi+CndA
+fmrFwFAfE0KYdrnFGfFf/kZXkM0h+0+vcwIcjxGTidp6GpIUV5dGDR6kZHQ738v
C3S78HAXbLQkPylaWsrTpiUUKpLXMESEP7VpkV/E3RZ+4kuQSVyg9422jg7xaMJA
vJMvYFVZ+rAxghiJGti3XQq+/QehWM5o8+I=
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIDUTCCAjmgAwIBAgIIWMEl+2yguMYwDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE
Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTY1OTExMTQzMDAe
Fw0yMzA4MjgxNjE3MjdaFw0yNTEwMjYxNjE3MjhaMDYxNDAyBgNVBAMMK29wZW5z
aGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE2NTkxMTE0MzAwggEiMA0GCSqG
SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDEzvQ+VySQK/k/0sKVdwN7J4E4OJ8h+9GC
rDS38cLnYD3q6I/iC3ZoIZkkCkcbnHSc0/4Q/AKecXsb4pwI+9WPE5w2YQmtY6ey
2VB6Bg1BYTLw65WsWmm0CjszjMFSxyn3spesKFlYuT8mepC9ynsSofUQFUrEHZk3
YSq6sz24+KXIzCZS3k7ECGqKSyNZg30jBZmqa8cPAaws/zl9/U/rXP994qsNFruQ
DcLO1IVHYl650oOT6zswNhlzZ311fNIbf0S8VzgVxiC+TQgQJ1NQar2NmpROMSgX
Ybw6dFRxodkFfcNQAGcrqWlPCQTxlGGrl5GW5IKjkIYanw5szD9HAgMBAAGjYzBh
MA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQeUF07
Q3vpPq2XGFc1v9xEqPZqADAfBgNVHSMEGDAWgBQeUF07Q3vpPq2XGFc1v9xEqPZq
ADANBgkqhkiG9w0BAQsFAAOCAQEAFgsXg4gciulG51Ls8W4mln4HDmYmrFLxwhZQ
qhYr0pK8p+/WHJ6wjQueMuUK2DRBX1IKnOcz3FbLgTssHp11tBxadQotVCzvaD+g
AV6njgdxIv4J0KIrONzMnlU31NkO9xRfXzyJHa6frZLxzIZ8glSiUY6U4q2Q6E9P
/eUQeVxoDthTV4iYzWBS/R3rnNBloB+2PAKUDNyNfnDwcA6f+Q4k818eI8cnbyaz
iumM/yE8V3pJfDdb1slZHEhEbR6T2DDDP7G0DOoCQ3sSbRwXQwSA2TRG/eVBBenZ
SDQgReolRpbl5pntsGPmNfmnJv7Wqwaqi3yWZQuvz0wVaH8Ilg==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIDUTCCAjmgAwIBAgIIVapBs0FkjS0wDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE
Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTY1OTExMTQzMDAe
Fw0yMzA4MjgxNjE3MjdaFw0yNTEwMjYxNjE3MjhaMDYxNDAyBgNVBAMMK29wZW5z
aGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE2NTkxMTE0MzAwggEiMA0GCSqG
SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDEzvQ+VySQK/k/0sKVdwN7J4E4OJ8h+9GC
rDS38cLnYD3q6I/iC3ZoIZkkCkcbnHSc0/4Q/AKecXsb4pwI+9WPE5w2YQmtY6ey
2VB6Bg1BYTLw65WsWmm0CjszjMFSxyn3spesKFlYuT8mepC9ynsSofUQFUrEHZk3
YSq6sz24+KXIzCZS3k7ECGqKSyNZg30jBZmqa8cPAaws/zl9/U/rXP994qsNFruQ
DcLO1IVHYl650oOT6zswNhlzZ311fNIbf0S8VzgVxiC+TQgQJ1NQar2NmpROMSgX
Ybw6dFRxodkFfcNQAGcrqWlPCQTxlGGrl5GW5IKjkIYanw5szD9HAgMBAAGjYzBh
MA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQeUF07
Q3vpPq2XGFc1v9xEqPZqADAfBgNVHSMEGDAWgBQcNt8uVjGPa7oUKkjbC0E2zBEp
PTANBgkqhkiG9w0BAQsFAAOCAQEAmnzSOU/eJbeYNFnrfuiEARnBm/vAsRU9yCgE
MgiGCVOHO//iOhCd2w0uQJCSUhk9yz0tvwofFfz5vqaSEk7IgY+BErlsA24/j6VY
6R7GshiCEZj/XeDIjwVhGRXnzOL+QadzVBbJVGFrr82LC5iw+x2bVmLoNs9VjKDN
MM8GaQgJ1PbpPm23GwRcpsdPRuvOFwkdVU+9hxMFqZsHEz1AqILbFqog7Z9Fh88O
s8FbK3nVYImTBQLCdjUlXZY7oNdex6NkB1v8UJgRgTUkXAr+y3j7yYfMqL0yrPLf
lsbsbfGJPClVboj28qZf7lkiaLlt3Ae+2bwvSZAvJKlfJJ33NQ==
-----END CERTIFICATE-----

In summary when an internal certificate rotation takes place Noobaa has the new certificate in its secrets but continues to use the old certificate (presumably from RAM) until a point when it gets restarted. Noobaa should be able to detect when the new certificate arrives and restart it's self in a storage safe way (bearing in mind this is s3 and is used by lots of apps) which incurs no downtime for s3. For reference Quay has the same issue and they have had to fix this as well.