Bug 1703240

Summary: oc adm must-gather does not capture audit logs.
Product: OpenShift Container Platform Reporter: Luis Sanchez <sanchezl>
Component: ocAssignee: Luis Sanchez <sanchezl>
Status: CLOSED ERRATA QA Contact: zhou ying <yinzhou>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.1.0CC: aos-bugs, deads, erich, jokerman, lmeyer, mfojtik, mifiedle, mmccomas, xxia
Target Milestone: ---   
Target Release: 4.1.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-06-04 10:48:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
oc adm must-gather output
none
gather container logs none

Description Luis Sanchez 2019-04-25 20:55:29 UTC
Description of problem:

oc adm must-gather does not capture audit logs.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Run `os adm must-gather`
2. Examine output
3.

Actual results:

No audit logs.

Expected results:

Audit logs.

Additional info:

Comment 1 Eric Rich 2019-04-25 21:27:12 UTC
I think we can resolve this with https://github.com/openshift/must-gather/pull/84

Comment 3 David Eads 2019-04-26 11:49:52 UTC
We need this to be able to debug failures we've seen in bugzillas so far.  Eric Rich has a pull in progress https://github.com/openshift/must-gather/pull/84

Comment 5 zhou ying 2019-04-29 10:30:13 UTC
Confirmed with latest ocp , the audit_logs dir has created , but didn't download the audit log from openshift-apiserver and kube-apiserver:

[yinzhou@192 must-gather.local.5597455149930097316]$ cd audit_logs/
[yinzhou@192 audit_logs]$ ll
total 8
drwxr-xr-x. 2 yinzhou yinzhou 4096 Apr 29 18:21 kube-apiserver
-rw-r--r--. 1 yinzhou yinzhou    0 Apr 29 18:20 kube-apiserver.audit_logs_listing
drwxr-xr-x. 2 yinzhou yinzhou 4096 Apr 29 18:21 openshift-apiserver
-rw-r--r--. 1 yinzhou yinzhou    0 Apr 29 18:20 openshift-apiserver.audit_logs_listing
[yinzhou@192 audit_logs]$ cd kube-apiserver/
[yinzhou@192 kube-apiserver]$ ll
total 0

[yinzhou@192 kube-apiserver]$ pwd
/home/yinzhou/Downloads/must-gather.local.5597455149930097316/audit_logs/kube-apiserver

[yinzhou@192 audit_logs]$ cd openshift-apiserver/
[yinzhou@192 openshift-apiserver]$ ll
total 0


[yinzhou@192 audit_logs]$ oc version 
Client Version: version.Info{Major:"4", Minor:"1+", GitVersion:"v4.1.0", GitCommit:"e8d1fd69b", GitTreeState:"clean", BuildDate:"2019-04-29T06:42:38Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.4+cdaca6f", GitCommit:"cdaca6f", GitTreeState:"clean", BuildDate:"2019-04-28T17:33:06Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}


Payload: 4.1.0-0.nightly-2019-04-28-233640

Comment 6 Maciej Szulik 2019-04-29 13:07:04 UTC
I've double checked that on 4.1.0-0.okd-2019-04-29-081740 and it's scraping properly audit logs. Moving back to QA, I'm guessing you didn't hit latest release.

Comment 7 Mike Fiedler 2019-05-03 02:06:39 UTC
Tested on # oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.1.0-0.nightly-2019-05-02-131943   True        False         7h37m   Cluster version is 4.1.0-0.nightly-2019-05-02-131943

oc adm must-gather output does not include audit logs .

1.   oc adm must-gather --dest-dir=<location>
2.   find . -name "audit.log"   -> nothing found
3.   grep -r <content known to exist in audit logs> -> nothing found

oc adm must-gather tarball attached.

Comment 8 Mike Fiedler 2019-05-03 02:07:37 UTC
Created attachment 1562103 [details]
oc adm must-gather output

Comment 9 Eric Rich 2019-05-03 13:16:01 UTC
(In reply to Mike Fiedler from comment #7)
> Tested on # oc get clusterversion
> NAME      VERSION                             AVAILABLE   PROGRESSING  
> SINCE   STATUS
> version   4.1.0-0.nightly-2019-05-02-131943   True        False        
> 7h37m   Cluster version is 4.1.0-0.nightly-2019-05-02-131943
> 
> oc adm must-gather output does not include audit logs .
> 
> 1.   oc adm must-gather --dest-dir=<location>
> 2.   find . -name "audit.log"   -> nothing found
> 3.   grep -r <content known to exist in audit logs> -> nothing found
> 
> oc adm must-gather tarball attached.

Do you have or can you get logs from the pod that collected these artifacts? 

> Probably not we delete it.

Comment 10 Luis Sanchez 2019-05-03 13:29:07 UTC
@Mike, 

We are unable to reproduce. Can you help us by gathering some more information:

1. Run must-gather, keeping the created resources:

   oc adm must-gather --keep`

2. Using the namespace from the output, get name of must-gather pod:

   oc get -n openshift-must-gather-xxxxx pod

3. Get the log for the `gather` container

  oc -n openshift-must-gather-xxxxx log must-gather-xxxxx -c gather 


thanks

Comment 11 Mike Fiedler 2019-05-03 17:08:48 UTC
Created attachment 1562608 [details]
gather container logs

WARNING: Collecting one or more audit logs on ALL masters in your cluster. This could take a large amount of time.
/usr/bin/gather_audit_logs: line 28: /usr/bin/oc: No such file or directory
INFO: Audit logs for openshift-apiserver collected.
WARNING: Collecting one or more audit logs on ALL masters in your cluster. This could take a large amount of time.
/usr/bin/gather_audit_logs: line 28: /usr/bin/oc: No such file or directory
INFO: Audit logs for kube-apiserver collected.


Problem with the container image?

Comment 12 Luis Sanchez 2019-05-03 19:13:57 UTC
Please provide the output of:

oc -n openshift describe is must-gather

also, please confirm the oc binary version.

You can also try to specify the image to use directly: 

oc adm must-gather --image quay.io/openshift/origin-must-gather:latest

Comment 13 Luis Sanchez 2019-05-03 19:23:48 UTC
Mike did provide some of the requested info via Slack, his must-gather imagestream is pointing to v4.0 images which are no longer being updated.

Comment 15 Luke Meyer 2019-05-03 21:10:42 UTC
All of the 4.1 images, and presumably releases after that, are going into ocp-v4.0-art-dev. That's how we expect to ship it.

They're all that way and they should all be up to date; see:
oc adm release info --pullspecs registry.svc.ci.openshift.org/ocp/release:4.1.0-0.nightly-2019-05-03-093152

The release tag is all customers will normally see.

Comment 16 Luis Sanchez 2019-05-07 18:46:41 UTC
Bug reproduced. 

Base images are specified in different locations for CI vs. OCP. Original update changing base from base to cli only changed base in CI builds.

PR opened: https://github.com/openshift/ocp-build-data/pull/116

Comment 17 David Eads 2019-05-07 18:48:31 UTC
> Original update changing base from base to cli only changed base in CI builds.

This is why the e2e testing for must-gather was all successful.

Comment 18 David Eads 2019-05-07 19:58:55 UTC
https://github.com/openshift/ocp-build-data/pull/116 will fix the issue in the next build.

Comment 19 Luis Sanchez 2019-05-08 12:36:53 UTC
Fix available starting with 4.1.0-0.nightly-2019-05-08-001504

Comment 20 zhou ying 2019-05-09 05:47:03 UTC
Confirmed with latest ocp version, the issue has fixed:
[root@dhcp-140-138 audit_logs]# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.1.0-0.nightly-2019-05-08-131137   True        False         114m    Cluster version is 4.1.0-0.nightly-2019-05-08-131137

[root@dhcp-140-138 must-gather.local.4873133549046800314]# find . -name "*audit.log*" 
./audit_logs/kube-apiserver/ip-172-31-156-157.ap-northeast-2.compute.internal-audit.log.gz
./audit_logs/kube-apiserver/ip-172-31-137-162.ap-northeast-2.compute.internal-audit.log.gz
./audit_logs/kube-apiserver/ip-172-31-137-134.ap-northeast-2.compute.internal-audit.log.gz
./audit_logs/openshift-apiserver/ip-172-31-156-157.ap-northeast-2.compute.internal-audit.log.gz
./audit_logs/openshift-apiserver/ip-172-31-137-162.ap-northeast-2.compute.internal-audit.log.gz
./audit_logs/openshift-apiserver/ip-172-31-137-134.ap-northeast-2.compute.internal-audit.log.gz

Comment 22 errata-xmlrpc 2019-06-04 10:48:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758