Bug 2021629 - api request counts for current hour are incorrect
Summary: api request counts for current hour are incorrect
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-apiserver
Version: 4.8
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.10.0
Assignee: Luis Sanchez
QA Contact: Ke Wang
URL:
Whiteboard:
: 2025524 (view as bug list)
Depends On:
Blocks: 2030697
TreeView+ depends on / blocked
 
Reported: 2021-11-09 19:22 UTC by Luis Sanchez
Modified: 2022-03-10 16:26 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-03-10 16:26:15 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift kubernetes pull 1037 0 None open Bug 2021629: UPSTREAM: <carry>: api request counts for current hour are incorrect 2021-11-09 19:52:01 UTC
Red Hat Knowledge Base (Article) 6329921 0 None None None 2021-12-14 16:45:08 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-10 16:26:37 UTC

Description Luis Sanchez 2021-11-09 19:22:32 UTC
Description of problem:

The currentHour field of APIRequestCount resources is sometimes incorrect. This makes it difficult for users checking for deprecated API usage before upgrading to tell if a deprecated API is in use.

Steps to Reproduce:
1. Install 4.7.x
2. Upgrade to 4.8.x

Actual results:

Hours after the upgrade has completed, numerous deprecated API requests are logged in `currentHour` even though the APIs are no longer in use.

Expected results:

The contents of `currentHour` should be correct.

Comment 1 Scott Dodson 2021-11-09 20:13:12 UTC
I've increased the severity and priority as it'd be great for us to get this fix into 4.8.z ASAP.

Comment 5 Ke Wang 2021-12-11 02:29:55 UTC
Verification steps,
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
On a 4.10 cluster, use a deprecated API,
$ oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.0-0.nightly-2021-12-10-033652   True        False         46m     Cluster version is 4.10.0-0.nightly-2021-12-10-033652

Get current apirequestcount of cronjobs.v1beta1.batch:
NAME                     REMOVEDINRELEASE   REQUESTSINCURRENTHOUR   REQUESTSINLAST24H
cronjobs.v1beta1.batch   1.25               0                       9

Try to access the deprecated API cronjobs.v1beta1.batch ...
$ oc get cronjobs.v1beta1.batch
W1210 12:25:04.832161    3544 warnings.go:70] batch/v1beta1 CronJob is deprecated in v1.21+, unavailable in v1.25+; use batch/v1 CronJob
No resources found in default namespace.

----------------------------------
After 1 hour(s) later, check the deprecated apirequestcount of cronjobs.v1beta1.batch, no longer has count > 0 for the deprecated API.
$ oc get apirequestcount/cronjobs.v1beta1.batch
NAME                     REMOVEDINRELEASE   REQUESTSINCURRENTHOUR   REQUESTSINLAST24H
cronjobs.v1beta1.batch   1.25               1                       23
----------------------------------
After 2 hour(s) later, check the deprecated apirequestcount of cronjobs.v1beta1.batch, no longer has count > 0 for the deprecated API.
$ oc get apirequestcount/cronjobs.v1beta1.batch
NAME                     REMOVEDINRELEASE   REQUESTSINCURRENTHOUR   REQUESTSINLAST24H
cronjobs.v1beta1.batch   1.25               1                       32

----------------------------------
Delete apiresourcecounts/batch.v1beta1.cronjobs . Observe that it does not get recreated. (wait at least 5 min)
Sat 11 Dec 2021 12:10:51 AM CST
$ oc delete apirequestcount/cronjobs.v1beta1.batch
apirequestcount.apiserver.openshift.io "cronjobs.v1beta1.batch" deleted
Sat 11 Dec 2021 12:20:54 AM CST
Ten minutes later, 
$ oc get apirequestcount | grep ‘cronjobs.v1beta1.batch’
No results found.

Invoke the deprecated API again and confirm apirequestcount/cronjobs.v1beta1.batch gets recreated.
Sat 11 Dec 2021 12:21:01 AM CST
$ oc get apirequestcount/cronjobs.v1beta1.batch
NAME                     REMOVEDINRELEASE   REQUESTSINCURRENTHOUR   REQUESTSINLAST24H
cronjobs.v1beta1.batch   1.25               1                       1

I have one question about REQUESTSINLAST24H which always keeps going up,if REQUESTSINCURRENTHOUR stops going up, why is the REQUESTSINLAST24H still increasing?

Comment 7 Ke Wang 2021-12-13 02:15:04 UTC
Per Comment 5, The PR fix has resolved the problem that the currentHour field of APIRequestCount resources is sometimes incorrect. So move the bug VERIFIED, We can go on back-porting to 4.8 release. Will track the problem REQUESTSINLAST24H still increasing in another thread.

Comment 8 Jack Ottofaro 2021-12-14 20:33:47 UTC
*** Bug 2025524 has been marked as a duplicate of this bug. ***

Comment 10 W. Trevor King 2021-12-23 05:57:43 UTC
Target release for this bug is 4.10.0.  For folks looking for backports, click the "Show advanced fields" button, and find the "Blocks" property.  Click through that blocks chain to find bug 2030697 (4.9.z) and from there on to bug 2032325 (4.8.z).

Comment 13 errata-xmlrpc 2022-03-10 16:26:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056


Note You need to log in before you can comment on or make changes to this bug.