Bug 1988828

Summary: oc adm must-gather runs successfully for audit logs 2e2 is failing
Product: OpenShift Container Platform Reporter: Eran Cohen <ercohen>
Component: ocAssignee: Eran Cohen <ercohen>
Status: CLOSED ERRATA QA Contact: zhou ying <yinzhou>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.9CC: aos-bugs, jokerman, mfojtik, rfreiman
Target Milestone: ---   
Target Release: 4.9.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: tag-ci
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-10-18 17:43:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
oauth-apiserver.audit_logs_listing none

Description Eran Cohen 2021-08-01 13:17:20 UTC 
Created attachment 1809909 [details]
oauth-apiserver.audit_logs_listing

Description of problem:
The test is failing due to `expected files should not be empty` error but if you download the audit logs you can see that the files are not empty. 

It seems that the test is validating the file has at least 50 bytes. but these files sometimes have less than that:

Example:

fail [github.com/openshift/origin/test/extended/cli/mustgather.go:290]: Unexpected error:
    <*errors.errorString | 0xc002417f20>: {
        s: "expected files should not be empty: /tmp/test.oc-adm-must-gather.617883005/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-77c10600473b2e33e4fdb5d3280972f646870fc686b02beebe488dbad73bb536/audit_logs/openshift-apiserver.audit_logs_listing,/tmp/test.oc-adm-must-gather.617883005/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-77c10600473b2e33e4fdb5d3280972f646870fc686b02beebe488dbad73bb536/audit_logs/oauth-apiserver.audit_logs_listing",
    }


Actual files content:

openshift-apiserver.audit_logs_listing:
ip-10-0-151-198.ec2.internal audit.log

oauth-apiserver.audit_logs_listing:

ip-10-0-151-198.ec2.internal audit.log



Version-Release number of selected component (if applicable):


How reproducible:
 18% in 4.9 SNO CI
 27% in 4.8 SNO CI
 https://search.ci.openshift.org/?search=oc+adm+must-gather+runs+successfully+for+audit+logs&maxAge=336h&context=-1&type=bug%2Bjunit&name=aws-single-node&excludeName=&maxMatches=1&maxBytes=20971520&groupBy=job

Steps to Reproduce:
1.Happens on CI and PR jobs
Example: https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.9-e2e-aws-single-node/1420900400336211968

Actual results:
Test is failing although the listed files are not empty

Expected results:
Expected the test to pass 

Additional info:
Attached oauth audit logs
Comment 1 Eran Cohen 2021-08-03 07:58:48 UTC 
This is the file content when the test pass (link):

ip-10-0-129-40.us-west-2.compute.internal audit.log


- 51 chars
Comment 4 zhou ying 2021-09-02 06:54:46 UTC 
Can't see the same error with follow search : 

https://search.ci.openshift.org/?search=oc+adm+must-gather+runs+successfully+for+audit+logs&maxAge=336h&context=-1&type=bug%2Bjunit&name=aws-single-node&excludeName=&maxMatches=1&maxBytes=20971520&groupBy=job
Comment 7 errata-xmlrpc 2021-10-18 17:43:44 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759