Bug 1959699

Summary: [RFE] Collect LSO pod log and daemonset log managed by LSO
Product: OpenShift Container Platform Reporter: Qin Ping <piqin>
Component: StorageAssignee: melbeher
Storage sub component: Local Storage Operator QA Contact: Chao Yang <chaoyang>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: unspecified CC: aos-bugs, chuffman, jsafrane
Version: 4.8   
Target Milestone: ---   
Target Release: 4.8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-07-27 23:08:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Qin Ping 2021-05-12 06:58:05 UTC 
Description of problem:
When run must gather cmd with common image or local must gather image, no LSO pod log and daemonsets log managed by LSO is collected.

Version-Release number of selected component (if applicable):
$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-05-12-002851   True        False         3h43m   Cluster version is 4.8.0-0.nightly-2021-05-12-002851

$ oc get csv -n openshift-local-storage
NAME                                           DISPLAY         VERSION                 REPLACES   PHASE
local-storage-operator.4.8.0-202105111002.p0   Local Storage   4.8.0-202105111002.p0              Succeeded


How reproducible:


Steps to Reproduce:
1. ran: oc adm must-gather
logs for LSO:
[piqin@preserve-storage-server1 openshift-local-storage]$ find .
.
./monitoring.coreos.com
./monitoring.coreos.com/servicemonitors
./monitoring.coreos.com/servicemonitors/local-storage-operator-metrics.yaml
./operators.coreos.com
./operators.coreos.com/clusterserviceversions
./operators.coreos.com/clusterserviceversions/local-storage-operator.4.8.0-202105111002.p0.yaml
./operators.coreos.com/installplans
./operators.coreos.com/installplans/install-mh2pf.yaml
./operators.coreos.com/installplans/install-mqlkh.yaml
./operators.coreos.com/operatorconditions
./operators.coreos.com/operatorconditions/local-storage-operator.4.8.0-202105111002.p0.yaml
./operators.coreos.com/operatorgroups
./operators.coreos.com/operatorgroups/openshift-local-storage-wbm8w.yaml
./operators.coreos.com/subscriptions
./operators.coreos.com/subscriptions/local-storage-operator.yaml
2. ran: $ oc adm must-gather --image=quay.io/openshift/origin-local-storage-mustgather:4.8
logs for LSO:
[piqin@preserve-storage-server1 openshift-local-storage]$ find .
.
./crs
./crs/localvolumesets.local.storage.openshift.io
./crs/localvolumesets.local.storage.openshift.io/lvs-test.yaml
3.

Actual results:


Expected results:

Master Log:

Node Log (of failed PODs):

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:
Comment 4 Chao Yang 2021-05-27 06:02:19 UTC 
find . | grep pods
./quay-io-openshift-origin-local-storage-mustgather-sha256-caf0e798f8927682289e3d97868976f6f6997bd3776575b847a5d8c3135cbf44/namespaces/openshift-local-storage/pods
./quay-io-openshift-origin-local-storage-mustgather-sha256-caf0e798f8927682289e3d97868976f6f6997bd3776575b847a5d8c3135cbf44/namespaces/openshift-local-storage/pods/diskmaker-manager-tvkgk.log
./quay-io-openshift-origin-local-storage-mustgather-sha256-caf0e798f8927682289e3d97868976f6f6997bd3776575b847a5d8c3135cbf44/namespaces/openshift-local-storage/pods/pod5.log
./quay-io-openshift-origin-local-storage-mustgather-sha256-caf0e798f8927682289e3d97868976f6f6997bd3776575b847a5d8c3135cbf44/namespaces/openshift-local-storage/pods/diskmaker-manager-f89pp.log
./quay-io-openshift-origin-local-storage-mustgather-sha256-caf0e798f8927682289e3d97868976f6f6997bd3776575b847a5d8c3135cbf44/namespaces/openshift-local-storage/pods/local-storage-operator-7cc8799669-lzcbg.log
./quay-io-openshift-origin-local-storage-mustgather-sha256-caf0e798f8927682289e3d97868976f6f6997bd3776575b847a5d8c3135cbf44/namespaces/openshift-local-storage/pods/diskmaker-manager-n9gvd.log

@chuffman I have one question for this fix.
It seems if install Local Storage Operator in other namespace created by customers, could not get the pods logs
Comment 5 Christian Huffman 2021-05-27 14:52:01 UTC 
Chao Yang,

You're correct. We had the logic included to detect the correct namespace, and were storing the pod logs appropriately; however, I didn't update the pod log collection to use the appropriate namespace. This is fixed in https://github.com/openshift/local-storage-operator/pull/241 .
Comment 7 Chao Yang 2021-05-31 07:08:06 UTC 
[must-gather-tcm7h] OUT ./
[must-gather-tcm7h] OUT namespaces/
[must-gather-tcm7h] OUT namespaces/local-storage/
[must-gather-tcm7h] OUT namespaces/local-storage/crs/
[must-gather-tcm7h] OUT namespaces/local-storage/crs/localvolumediscoveries.local.storage.openshift.io/
[must-gather-tcm7h] OUT namespaces/local-storage/crs/localvolumediscoveries.local.storage.openshift.io/auto-discover-devices.yaml
[must-gather-tcm7h] OUT namespaces/local-storage/crs/localvolumediscoveryresults.local.storage.openshift.io/
[must-gather-tcm7h] OUT namespaces/local-storage/crs/localvolumediscoveryresults.local.storage.openshift.io/discovery-result-39920dce13c39ac6fcf697b0837765e1.yaml
[must-gather-tcm7h] OUT namespaces/local-storage/crs/localvolumediscoveryresults.local.storage.openshift.io/discovery-result-c03b3c4cb10ef9ed7f8a5c43c6bf7168.yaml
[must-gather-tcm7h] OUT namespaces/local-storage/crs/localvolumediscoveryresults.local.storage.openshift.io/discovery-result-d32af5f9fcfa145f216fbdf0cfff3395.yaml
[must-gather-tcm7h] OUT namespaces/local-storage/pods/
[must-gather-tcm7h] OUT namespaces/local-storage/pods/diskmaker-discovery-4qjpp.log
[must-gather-tcm7h] OUT namespaces/local-storage/pods/diskmaker-discovery-5wjcx.log
Comment 10 errata-xmlrpc 2021-07-27 23:08:01 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438