Bug 1959699 - [RFE] Collect LSO pod log and daemonset log managed by LSO
Summary: [RFE] Collect LSO pod log and daemonset log managed by LSO
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 4.8
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 4.8.0
Assignee: melbeher
QA Contact: Chao Yang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-05-12 06:58 UTC by Qin Ping
Modified: 2021-07-27 23:08 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-07-27 23:08:01 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift local-storage-operator pull 236 0 None open Bug 1959699: Updated gather script to collect pod logs 2021-05-17 14:23:34 UTC
Github openshift local-storage-operator pull 238 0 None open Bug 1959699: Adjusts command used to get pod information 2021-05-21 17:54:47 UTC
Github openshift local-storage-operator pull 241 0 None open Bug 1959699: Use correct namespace for must-gather 2021-05-27 14:49:39 UTC
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 23:08:18 UTC

Description Qin Ping 2021-05-12 06:58:05 UTC
Description of problem:
When run must gather cmd with common image or local must gather image, no LSO pod log and daemonsets log managed by LSO is collected.

Version-Release number of selected component (if applicable):
$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-05-12-002851   True        False         3h43m   Cluster version is 4.8.0-0.nightly-2021-05-12-002851

$ oc get csv -n openshift-local-storage
NAME                                           DISPLAY         VERSION                 REPLACES   PHASE
local-storage-operator.4.8.0-202105111002.p0   Local Storage   4.8.0-202105111002.p0              Succeeded


How reproducible:


Steps to Reproduce:
1. ran: oc adm must-gather
logs for LSO:
[piqin@preserve-storage-server1 openshift-local-storage]$ find .
.
./monitoring.coreos.com
./monitoring.coreos.com/servicemonitors
./monitoring.coreos.com/servicemonitors/local-storage-operator-metrics.yaml
./operators.coreos.com
./operators.coreos.com/clusterserviceversions
./operators.coreos.com/clusterserviceversions/local-storage-operator.4.8.0-202105111002.p0.yaml
./operators.coreos.com/installplans
./operators.coreos.com/installplans/install-mh2pf.yaml
./operators.coreos.com/installplans/install-mqlkh.yaml
./operators.coreos.com/operatorconditions
./operators.coreos.com/operatorconditions/local-storage-operator.4.8.0-202105111002.p0.yaml
./operators.coreos.com/operatorgroups
./operators.coreos.com/operatorgroups/openshift-local-storage-wbm8w.yaml
./operators.coreos.com/subscriptions
./operators.coreos.com/subscriptions/local-storage-operator.yaml
2. ran: $ oc adm must-gather --image=quay.io/openshift/origin-local-storage-mustgather:4.8
logs for LSO:
[piqin@preserve-storage-server1 openshift-local-storage]$ find .
.
./crs
./crs/localvolumesets.local.storage.openshift.io
./crs/localvolumesets.local.storage.openshift.io/lvs-test.yaml
3.

Actual results:


Expected results:

Master Log:

Node Log (of failed PODs):

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:

Comment 4 Chao Yang 2021-05-27 06:02:19 UTC
find . | grep pods
./quay-io-openshift-origin-local-storage-mustgather-sha256-caf0e798f8927682289e3d97868976f6f6997bd3776575b847a5d8c3135cbf44/namespaces/openshift-local-storage/pods
./quay-io-openshift-origin-local-storage-mustgather-sha256-caf0e798f8927682289e3d97868976f6f6997bd3776575b847a5d8c3135cbf44/namespaces/openshift-local-storage/pods/diskmaker-manager-tvkgk.log
./quay-io-openshift-origin-local-storage-mustgather-sha256-caf0e798f8927682289e3d97868976f6f6997bd3776575b847a5d8c3135cbf44/namespaces/openshift-local-storage/pods/pod5.log
./quay-io-openshift-origin-local-storage-mustgather-sha256-caf0e798f8927682289e3d97868976f6f6997bd3776575b847a5d8c3135cbf44/namespaces/openshift-local-storage/pods/diskmaker-manager-f89pp.log
./quay-io-openshift-origin-local-storage-mustgather-sha256-caf0e798f8927682289e3d97868976f6f6997bd3776575b847a5d8c3135cbf44/namespaces/openshift-local-storage/pods/local-storage-operator-7cc8799669-lzcbg.log
./quay-io-openshift-origin-local-storage-mustgather-sha256-caf0e798f8927682289e3d97868976f6f6997bd3776575b847a5d8c3135cbf44/namespaces/openshift-local-storage/pods/diskmaker-manager-n9gvd.log

@chuffman I have one question for this fix.
It seems if install Local Storage Operator in other namespace created by customers, could not get the pods logs

Comment 5 Christian Huffman 2021-05-27 14:52:01 UTC
Chao Yang,

You're correct. We had the logic included to detect the correct namespace, and were storing the pod logs appropriately; however, I didn't update the pod log collection to use the appropriate namespace. This is fixed in https://github.com/openshift/local-storage-operator/pull/241 .

Comment 7 Chao Yang 2021-05-31 07:08:06 UTC
[must-gather-tcm7h] OUT ./
[must-gather-tcm7h] OUT namespaces/
[must-gather-tcm7h] OUT namespaces/local-storage/
[must-gather-tcm7h] OUT namespaces/local-storage/crs/
[must-gather-tcm7h] OUT namespaces/local-storage/crs/localvolumediscoveries.local.storage.openshift.io/
[must-gather-tcm7h] OUT namespaces/local-storage/crs/localvolumediscoveries.local.storage.openshift.io/auto-discover-devices.yaml
[must-gather-tcm7h] OUT namespaces/local-storage/crs/localvolumediscoveryresults.local.storage.openshift.io/
[must-gather-tcm7h] OUT namespaces/local-storage/crs/localvolumediscoveryresults.local.storage.openshift.io/discovery-result-39920dce13c39ac6fcf697b0837765e1.yaml
[must-gather-tcm7h] OUT namespaces/local-storage/crs/localvolumediscoveryresults.local.storage.openshift.io/discovery-result-c03b3c4cb10ef9ed7f8a5c43c6bf7168.yaml
[must-gather-tcm7h] OUT namespaces/local-storage/crs/localvolumediscoveryresults.local.storage.openshift.io/discovery-result-d32af5f9fcfa145f216fbdf0cfff3395.yaml
[must-gather-tcm7h] OUT namespaces/local-storage/pods/
[must-gather-tcm7h] OUT namespaces/local-storage/pods/diskmaker-discovery-4qjpp.log
[must-gather-tcm7h] OUT namespaces/local-storage/pods/diskmaker-discovery-5wjcx.log

Comment 10 errata-xmlrpc 2021-07-27 23:08:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438


Note You need to log in before you can comment on or make changes to this bug.