Bug 2021629

Summary: api request counts for current hour are incorrect
Product: OpenShift Container Platform Reporter: Luis Sanchez <sanchezl>
Component: kube-apiserverAssignee: Luis Sanchez <sanchezl>
Status: CLOSED ERRATA QA Contact: Ke Wang <kewang>
Severity: high Docs Contact:
Priority: high    
Version: 4.8CC: andbartl, anmistry, aos-bugs, kewang, mfojtik, oarribas, rsandu, wking, xxia
Target Milestone: ---Keywords: FastFix
Target Release: 4.10.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-03-10 16:26:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2030697    

Description Luis Sanchez 2021-11-09 19:22:32 UTC
Description of problem:

The currentHour field of APIRequestCount resources is sometimes incorrect. This makes it difficult for users checking for deprecated API usage before upgrading to tell if a deprecated API is in use.

Steps to Reproduce:
1. Install 4.7.x
2. Upgrade to 4.8.x

Actual results:

Hours after the upgrade has completed, numerous deprecated API requests are logged in `currentHour` even though the APIs are no longer in use.

Expected results:

The contents of `currentHour` should be correct.

Comment 1 Scott Dodson 2021-11-09 20:13:12 UTC
I've increased the severity and priority as it'd be great for us to get this fix into 4.8.z ASAP.

Comment 5 Ke Wang 2021-12-11 02:29:55 UTC
Verification steps,
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
On a 4.10 cluster, use a deprecated API,
$ oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.0-0.nightly-2021-12-10-033652   True        False         46m     Cluster version is 4.10.0-0.nightly-2021-12-10-033652

Get current apirequestcount of cronjobs.v1beta1.batch:
NAME                     REMOVEDINRELEASE   REQUESTSINCURRENTHOUR   REQUESTSINLAST24H
cronjobs.v1beta1.batch   1.25               0                       9

Try to access the deprecated API cronjobs.v1beta1.batch ...
$ oc get cronjobs.v1beta1.batch
W1210 12:25:04.832161    3544 warnings.go:70] batch/v1beta1 CronJob is deprecated in v1.21+, unavailable in v1.25+; use batch/v1 CronJob
No resources found in default namespace.

----------------------------------
After 1 hour(s) later, check the deprecated apirequestcount of cronjobs.v1beta1.batch, no longer has count > 0 for the deprecated API.
$ oc get apirequestcount/cronjobs.v1beta1.batch
NAME                     REMOVEDINRELEASE   REQUESTSINCURRENTHOUR   REQUESTSINLAST24H
cronjobs.v1beta1.batch   1.25               1                       23
----------------------------------
After 2 hour(s) later, check the deprecated apirequestcount of cronjobs.v1beta1.batch, no longer has count > 0 for the deprecated API.
$ oc get apirequestcount/cronjobs.v1beta1.batch
NAME                     REMOVEDINRELEASE   REQUESTSINCURRENTHOUR   REQUESTSINLAST24H
cronjobs.v1beta1.batch   1.25               1                       32

----------------------------------
Delete apiresourcecounts/batch.v1beta1.cronjobs . Observe that it does not get recreated. (wait at least 5 min)
Sat 11 Dec 2021 12:10:51 AM CST
$ oc delete apirequestcount/cronjobs.v1beta1.batch
apirequestcount.apiserver.openshift.io "cronjobs.v1beta1.batch" deleted
Sat 11 Dec 2021 12:20:54 AM CST
Ten minutes later, 
$ oc get apirequestcount | grep ‘cronjobs.v1beta1.batch’
No results found.

Invoke the deprecated API again and confirm apirequestcount/cronjobs.v1beta1.batch gets recreated.
Sat 11 Dec 2021 12:21:01 AM CST
$ oc get apirequestcount/cronjobs.v1beta1.batch
NAME                     REMOVEDINRELEASE   REQUESTSINCURRENTHOUR   REQUESTSINLAST24H
cronjobs.v1beta1.batch   1.25               1                       1

I have one question about REQUESTSINLAST24H which always keeps going up,if REQUESTSINCURRENTHOUR stops going up, why is the REQUESTSINLAST24H still increasing?

Comment 7 Ke Wang 2021-12-13 02:15:04 UTC
Per Comment 5, The PR fix has resolved the problem that the currentHour field of APIRequestCount resources is sometimes incorrect. So move the bug VERIFIED, We can go on back-porting to 4.8 release. Will track the problem REQUESTSINLAST24H still increasing in another thread.

Comment 8 Jack Ottofaro 2021-12-14 20:33:47 UTC
*** Bug 2025524 has been marked as a duplicate of this bug. ***

Comment 10 W. Trevor King 2021-12-23 05:57:43 UTC
Target release for this bug is 4.10.0.  For folks looking for backports, click the "Show advanced fields" button, and find the "Blocks" property.  Click through that blocks chain to find bug 2030697 (4.9.z) and from there on to bug 2032325 (4.8.z).

Comment 13 errata-xmlrpc 2022-03-10 16:26:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056