Hide Forgot
Description of problem: oc adm must-gather does not capture audit logs. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. Run `os adm must-gather` 2. Examine output 3. Actual results: No audit logs. Expected results: Audit logs. Additional info:
I think we can resolve this with https://github.com/openshift/must-gather/pull/84
We need this to be able to debug failures we've seen in bugzillas so far. Eric Rich has a pull in progress https://github.com/openshift/must-gather/pull/84
Confirmed with latest ocp , the audit_logs dir has created , but didn't download the audit log from openshift-apiserver and kube-apiserver: [yinzhou@192 must-gather.local.5597455149930097316]$ cd audit_logs/ [yinzhou@192 audit_logs]$ ll total 8 drwxr-xr-x. 2 yinzhou yinzhou 4096 Apr 29 18:21 kube-apiserver -rw-r--r--. 1 yinzhou yinzhou 0 Apr 29 18:20 kube-apiserver.audit_logs_listing drwxr-xr-x. 2 yinzhou yinzhou 4096 Apr 29 18:21 openshift-apiserver -rw-r--r--. 1 yinzhou yinzhou 0 Apr 29 18:20 openshift-apiserver.audit_logs_listing [yinzhou@192 audit_logs]$ cd kube-apiserver/ [yinzhou@192 kube-apiserver]$ ll total 0 [yinzhou@192 kube-apiserver]$ pwd /home/yinzhou/Downloads/must-gather.local.5597455149930097316/audit_logs/kube-apiserver [yinzhou@192 audit_logs]$ cd openshift-apiserver/ [yinzhou@192 openshift-apiserver]$ ll total 0 [yinzhou@192 audit_logs]$ oc version Client Version: version.Info{Major:"4", Minor:"1+", GitVersion:"v4.1.0", GitCommit:"e8d1fd69b", GitTreeState:"clean", BuildDate:"2019-04-29T06:42:38Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.4+cdaca6f", GitCommit:"cdaca6f", GitTreeState:"clean", BuildDate:"2019-04-28T17:33:06Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"} Payload: 4.1.0-0.nightly-2019-04-28-233640
I've double checked that on 4.1.0-0.okd-2019-04-29-081740 and it's scraping properly audit logs. Moving back to QA, I'm guessing you didn't hit latest release.
Tested on # oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.1.0-0.nightly-2019-05-02-131943 True False 7h37m Cluster version is 4.1.0-0.nightly-2019-05-02-131943 oc adm must-gather output does not include audit logs . 1. oc adm must-gather --dest-dir=<location> 2. find . -name "audit.log" -> nothing found 3. grep -r <content known to exist in audit logs> -> nothing found oc adm must-gather tarball attached.
Created attachment 1562103 [details] oc adm must-gather output
(In reply to Mike Fiedler from comment #7) > Tested on # oc get clusterversion > NAME VERSION AVAILABLE PROGRESSING > SINCE STATUS > version 4.1.0-0.nightly-2019-05-02-131943 True False > 7h37m Cluster version is 4.1.0-0.nightly-2019-05-02-131943 > > oc adm must-gather output does not include audit logs . > > 1. oc adm must-gather --dest-dir=<location> > 2. find . -name "audit.log" -> nothing found > 3. grep -r <content known to exist in audit logs> -> nothing found > > oc adm must-gather tarball attached. Do you have or can you get logs from the pod that collected these artifacts? > Probably not we delete it.
@Mike, We are unable to reproduce. Can you help us by gathering some more information: 1. Run must-gather, keeping the created resources: oc adm must-gather --keep` 2. Using the namespace from the output, get name of must-gather pod: oc get -n openshift-must-gather-xxxxx pod 3. Get the log for the `gather` container oc -n openshift-must-gather-xxxxx log must-gather-xxxxx -c gather thanks
Created attachment 1562608 [details] gather container logs WARNING: Collecting one or more audit logs on ALL masters in your cluster. This could take a large amount of time. /usr/bin/gather_audit_logs: line 28: /usr/bin/oc: No such file or directory INFO: Audit logs for openshift-apiserver collected. WARNING: Collecting one or more audit logs on ALL masters in your cluster. This could take a large amount of time. /usr/bin/gather_audit_logs: line 28: /usr/bin/oc: No such file or directory INFO: Audit logs for kube-apiserver collected. Problem with the container image?
Please provide the output of: oc -n openshift describe is must-gather also, please confirm the oc binary version. You can also try to specify the image to use directly: oc adm must-gather --image quay.io/openshift/origin-must-gather:latest
Mike did provide some of the requested info via Slack, his must-gather imagestream is pointing to v4.0 images which are no longer being updated.
All of the 4.1 images, and presumably releases after that, are going into ocp-v4.0-art-dev. That's how we expect to ship it. They're all that way and they should all be up to date; see: oc adm release info --pullspecs registry.svc.ci.openshift.org/ocp/release:4.1.0-0.nightly-2019-05-03-093152 The release tag is all customers will normally see.
Bug reproduced. Base images are specified in different locations for CI vs. OCP. Original update changing base from base to cli only changed base in CI builds. PR opened: https://github.com/openshift/ocp-build-data/pull/116
> Original update changing base from base to cli only changed base in CI builds. This is why the e2e testing for must-gather was all successful.
https://github.com/openshift/ocp-build-data/pull/116 will fix the issue in the next build.
Fix available starting with 4.1.0-0.nightly-2019-05-08-001504
Confirmed with latest ocp version, the issue has fixed: [root@dhcp-140-138 audit_logs]# oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.1.0-0.nightly-2019-05-08-131137 True False 114m Cluster version is 4.1.0-0.nightly-2019-05-08-131137 [root@dhcp-140-138 must-gather.local.4873133549046800314]# find . -name "*audit.log*" ./audit_logs/kube-apiserver/ip-172-31-156-157.ap-northeast-2.compute.internal-audit.log.gz ./audit_logs/kube-apiserver/ip-172-31-137-162.ap-northeast-2.compute.internal-audit.log.gz ./audit_logs/kube-apiserver/ip-172-31-137-134.ap-northeast-2.compute.internal-audit.log.gz ./audit_logs/openshift-apiserver/ip-172-31-156-157.ap-northeast-2.compute.internal-audit.log.gz ./audit_logs/openshift-apiserver/ip-172-31-137-162.ap-northeast-2.compute.internal-audit.log.gz ./audit_logs/openshift-apiserver/ip-172-31-137-134.ap-northeast-2.compute.internal-audit.log.gz
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0758