Bug 2266550

Summary:	must-gather command with --dr flag is throwing error on IBM Power cluster
Product:	[Red Hat Storage] Red Hat OpenShift Data Foundation	Reporter:	Aaruni Aggarwal <aaaggarw>
Component:	must-gather	Assignee:	yati padia <ypadia>
Status:	ASSIGNED ---	QA Contact:	Prasad Desala <tdesala>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	4.15	CC:	muagarwa, ngowda, odf-bz-bot, prsurve, ypadia
Target Milestone:	---	Flags:	ypadia: needinfo? (aaaggarw)
Target Release:	---
Hardware:	ppc64le
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Aaruni Aggarwal 2024-02-28 08:09:37 UTC

Description of problem (please be detailed as possible and provide log
snippests):

While validating https://issues.redhat.com/browse/RHSTOR-4814, Found following:
must-gather command with --dr/ -d flag on RDR enabled cluster is throwing an error on IBM Power, although details are getting collected.

Version of all relevant components (if applicable):
OCP: 4.15.0-rc.5
ODF: 4.15.0-149.stable
MCO and DR hub operator on hub cluster: 4.15.0-149.stable
ACM: 2.10.0-79

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
No

Is there any workaround available to the best of your knowledge?
No

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?

Can this issue reproducible?
Yes

Can this issue reproduce from the UI?

If this is a regression, please provide more details to justify this:

Steps to Reproduce:
1. Create an OCP cluster
2. Install ACM and import managed clusters. Install Submariner-addons
3. Install Multicluster Orchestrator
4. Install ODF, and LSO and create a Storage system on both the managed clusters.
5. Create ApplicationSet.
6. Execute command on hub cluster:
oc adm must-gather --image=quay.io/rhceph-dev/ocs-must-gather:latest-4.15 -- /usr/bin/gather --dr

Actual results:
The following error is seen during the execution of the must-gather command with --dr flag.

error: arguments in resource/name form must have a single resource and name

Expected results:
No error should be there.

Additional info:

Comment 2 Aaruni Aggarwal 2024-02-28 08:11:51 UTC

Logs are getting collected but an error is seen. 

[root@rdr-hub-bastion-0 ~]# oc adm must-gather --image=quay.io/rhceph-dev/ocs-must-gather:latest-4.15 -- /usr/bin/gather --dr
[must-gather      ] OUT Using must-gather plug-in image: quay.io/rhceph-dev/ocs-must-gather:latest-4.15
When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information:
ClusterID: ed9b1001-7d73-4406-abce-d674408ea07b
ClientVersion: 4.15.0-rc.5
ClusterVersion: Stable at "4.15.0-rc.5"
ClusterOperators:
	All healthy and stable


[must-gather      ] OUT namespace/openshift-must-gather-mq5hb created
[must-gather      ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-p82hc created
[must-gather      ] OUT pod for plug-in image quay.io/rhceph-dev/ocs-must-gather:latest-4.15 created
[must-gather-csc6d] POD 2024-02-28T05:16:24.001029906Z volume percentage checker started.....
[must-gather-csc6d] POD 2024-02-28T05:16:24.012027645Z volume usage percentage 0
[must-gather-csc6d] POD 2024-02-28T05:16:24.020978569Z collection started at: 05:16:24 AM
[must-gather-csc6d] POD 2024-02-28T05:16:25.347977948Z error: the server doesn't have a resource type "storagecluster"
[must-gather-csc6d] POD 2024-02-28T05:16:25.513401235Z error: the server doesn't have a resource type "storagecluster"
[must-gather-csc6d] POD 2024-02-28T05:16:25.517196502Z checking for existing must-gather resource
[must-gather-csc6d] POD 2024-02-28T05:16:25.609407427Z No resources found in openshift-must-gather-mq5hb namespace.
[must-gather-csc6d] POD 2024-02-28T05:16:25.708182905Z not creating helper pod since storagecluster is not present
[must-gather-csc6d] POD 2024-02-28T05:16:25.708542460Z debugging node worker-0
[must-gather-csc6d] POD 2024-02-28T05:16:25.708714900Z debugging node worker-1
[must-gather-csc6d] POD 2024-02-28T05:16:25.708895646Z debugging node worker-2
[must-gather-csc6d] POD 2024-02-28T05:16:25.712589731Z waiting for 161 162 164 to terminate
[must-gather-csc6d] POD 2024-02-28T05:16:25.923392818Z Starting pod/worker-0-debug ...
[must-gather-csc6d] POD 2024-02-28T05:16:25.923392818Z To use host binaries, run `chroot /host`
[must-gather-csc6d] POD 2024-02-28T05:16:25.930243714Z Starting pod/worker-2-debug ...
[must-gather-csc6d] POD 2024-02-28T05:16:25.930243714Z To use host binaries, run `chroot /host`
[must-gather-csc6d] POD 2024-02-28T05:16:25.935244164Z Starting pod/worker-1-debug ...
[must-gather-csc6d] POD 2024-02-28T05:16:25.935244164Z To use host binaries, run `chroot /host`
[must-gather-csc6d] POD 2024-02-28T05:16:29.020683188Z volume usage percentage 0
[must-gather-csc6d] POD 2024-02-28T05:16:34.027649103Z volume usage percentage 0
[must-gather-csc6d] POD 2024-02-28T05:16:39.034348157Z volume usage percentage 0
[must-gather-csc6d] POD 2024-02-28T05:16:44.042110573Z volume usage percentage 0
[must-gather-csc6d] POD 2024-02-28T05:16:49.049308295Z volume usage percentage 0
[must-gather-csc6d] POD 2024-02-28T05:16:54.055883946Z volume usage percentage 0
[must-gather-csc6d] POD 2024-02-28T05:16:59.062819932Z volume usage percentage 0
[must-gather-csc6d] POD 2024-02-28T05:17:04.069206920Z volume usage percentage 0
[must-gather-csc6d] POD 2024-02-28T05:17:09.076626217Z volume usage percentage 0
[must-gather-csc6d] POD 2024-02-28T05:17:14.083122410Z volume usage percentage 0
[must-gather-csc6d] POD 2024-02-28T05:17:19.090244425Z volume usage percentage 0
[must-gather-csc6d] POD 2024-02-28T05:17:24.097572488Z volume usage percentage 0
[must-gather-csc6d] POD 2024-02-28T05:17:25.975677202Z error: arguments in resource/name form must have a single resource and name
[must-gather-csc6d] POD 2024-02-28T05:17:25.978322701Z error: arguments in resource/name form must have a single resource and name
[must-gather-csc6d] POD 2024-02-28T05:17:25.994041561Z error: arguments in resource/name form must have a single resource and name
[must-gather-csc6d] POD 2024-02-28T05:17:26.226733442Z error: resource name may not be empty
[must-gather-csc6d] POD 2024-02-28T05:17:26.238171022Z error: resource name may not be empty
[must-gather-csc6d] POD 2024-02-28T05:17:26.239396266Z error: resource name may not be empty
[must-gather-csc6d] POD 2024-02-28T05:17:26.241510376Z Collect DR logs...
[must-gather-csc6d] POD 2024-02-28T05:17:26.242053662Z Waiting on subprocesses to finish execution.
[must-gather-csc6d] POD 2024-02-28T05:17:26.374582977Z collecting dump of openshift-dr-system namespace
[must-gather-csc6d] POD 2024-02-28T05:17:29.104575547Z volume usage percentage 0
[must-gather-csc6d] POD 2024-02-28T05:17:34.111673791Z volume usage percentage 0
[must-gather-csc6d] POD 2024-02-28T05:17:34.956782377Z Gathering data for ns/openshift-operators...
[must-gather-csc6d] POD 2024-02-28T05:17:34.956782377Z I0228 05:17:28.023428     563 request.go:665] Waited for 1.182513937s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/apps.open-cluster-management.io/v1beta1?timeout=32s
[must-gather-csc6d] POD 2024-02-28T05:17:34.956782377Z W0228 05:17:34.538224     563 warnings.go:70] apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
[must-gather-csc6d] POD 2024-02-28T05:17:34.956782377Z Wrote inspect data to must-gather.
[must-gather-csc6d] POD 2024-02-28T05:17:34.960927541Z collecting oc command resource
[must-gather-csc6d] POD 2024-02-28T05:17:35.065833229Z collecting oc command resource
[must-gather-csc6d] POD 2024-02-28T05:17:35.172007935Z collecting oc command resource
[must-gather-csc6d] POD 2024-02-28T05:17:35.277026657Z collecting oc command resource
[must-gather-csc6d] POD 2024-02-28T05:17:35.338562850Z You must specify the type of resource to get. Use "oc api-resources" for a complete list of supported resources.
[must-gather-csc6d] POD 2024-02-28T05:17:35.338562850Z
[must-gather-csc6d] POD 2024-02-28T05:17:35.338614414Z error: Required resource not specified.
[must-gather-csc6d] POD 2024-02-28T05:17:35.338614414Z Use "oc explain <resource>" for a detailed description of that resource (e.g. oc explain pods).
[must-gather-csc6d] POD 2024-02-28T05:17:35.338614414Z See 'oc get -h' for help and examples
[must-gather-csc6d] POD 2024-02-28T05:17:35.342196520Z collecting oc describe configmap -n openshift-operators
[must-gather-csc6d] POD 2024-02-28T05:17:36.481603720Z error: the server doesn't have a resource type "storagecluster"
[must-gather-csc6d] POD 2024-02-28T05:17:36.651888826Z error: the server doesn't have a resource type "storagecluster"
[must-gather-csc6d] POD 2024-02-28T05:17:36.965870731Z pod "worker-0-debug" deleted
[must-gather-csc6d] POD 2024-02-28T05:17:37.478817344Z
[must-gather-csc6d] POD 2024-02-28T05:17:37.478817344Z Removing debug pod ...
[must-gather-csc6d] POD 2024-02-28T05:17:37.680840083Z pod "worker-1-debug" deleted
[must-gather-csc6d] POD 2024-02-28T05:17:38.623405715Z
[must-gather-csc6d] POD 2024-02-28T05:17:38.623405715Z Removing debug pod ...
[must-gather-csc6d] POD 2024-02-28T05:17:38.839046465Z pod "worker-2-debug" deleted
[must-gather-csc6d] POD 2024-02-28T05:17:39.119572268Z volume usage percentage 0
[must-gather-csc6d] POD 2024-02-28T05:17:40.193688492Z
[must-gather-csc6d] POD 2024-02-28T05:17:40.193688492Z Removing debug pod ...
[must-gather-csc6d] OUT waiting for gather to complete
[must-gather-csc6d] OUT downloading gather output
[must-gather-csc6d] OUT receiving incremental file list
[must-gather-csc6d] OUT ./
[must-gather-csc6d] OUT event-filter.html
[must-gather-csc6d] OUT gather-debug.log
[must-gather-csc6d] OUT timestamp
[must-gather-csc6d] OUT namespaces/
[must-gather-csc6d] OUT namespaces/openshift-operators/
[must-gather-csc6d] OUT namespaces/openshift-operators/openshift-operators.yaml
[must-gather-csc6d] OUT namespaces/openshift-operators/app.k8s.io/
[must-gather-csc6d] OUT namespaces/openshift-operators/app.k8s.io/applications.yaml
[must-gather-csc6d] OUT namespaces/openshift-operators/apps.openshift.io/
[must-gather-csc6d] OUT namespaces/openshift-operators/apps.openshift.io/deploymentconfigs.yaml
[must-gather-csc6d] OUT namespaces/openshift-operators/apps/
[must-gather-csc6d] OUT namespaces/openshift-operators/apps/daemonsets.yaml
[must-gather-csc6d] OUT namespaces/openshift-operators/apps/deployments.yaml
[must-gather-csc6d] OUT namespaces/openshift-operators/apps/replicasets.yaml
[must-gather-csc6d] OUT namespaces/openshift-operators/apps/statefulsets.yaml
[must-gather-csc6d] OUT namespaces/openshift-operators/autoscaling/
[must-gather-csc6d] OUT namespaces/openshift-operators/autoscaling/horizontalpodautoscalers.yaml
[must-gather-csc6d] OUT namespaces/openshift-operators/batch/
[must-gather-csc6d] OUT namespaces/openshift-operators/batch/cronjobs.yaml
[must-gather-csc6d] OUT namespaces/openshift-operators/batch/jobs.yaml
[must-gather-csc6d] OUT namespaces/openshift-operators/build.openshift.io/
[must-gather-csc6d] OUT namespaces/openshift-operators/build.openshift.io/buildconfigs.yaml
[must-gather-csc6d] OUT namespaces/openshift-operators/build.openshift.io/builds.yaml
[must-gather-csc6d] OUT namespaces/openshift-operators/core/
[must-gather-csc6d] OUT namespaces/openshift-operators/core/configmaps.yaml
[must-gather-csc6d] OUT namespaces/openshift-operators/core/endpoints.yaml
[must-gather-csc6d] OUT namespaces/openshift-operators/core/events.yaml
[must-gather-csc6d] OUT namespaces/openshift-operators/core/persistentvolumeclaims.yaml
[must-gather-csc6d] OUT namespaces/openshift-operators/core/pods.yaml
[must-gather-csc6d] OUT namespaces/openshift-operators/core/replicationcontrollers.yaml
[must-gather-csc6d] OUT namespaces/openshift-operators/core/secrets.yaml
[must-gather-csc6d] OUT namespaces/openshift-operators/core/services.yaml
[must-gather-csc6d] OUT namespaces/openshift-operators/discovery.k8s.io/
[must-gather-csc6d] OUT namespaces/openshift-operators/discovery.k8s.io/endpointslices.yaml
[must-gather-csc6d] OUT namespaces/openshift-operators/image.openshift.io/
[must-gather-csc6d] OUT namespaces/openshift-operators/image.openshift.io/imagestreams.yaml
[must-gather-csc6d] OUT namespaces/openshift-operators/k8s.ovn.org/
[must-gather-csc6d] OUT namespaces/openshift-operators/k8s.ovn.org/egressfirewalls.yaml
[must-gather-csc6d] OUT namespaces/openshift-operators/networking.k8s.io/
[must-gather-csc6d] OUT namespaces/openshift-operators/networking.k8s.io/networkpolicies.yaml
[must-gather-csc6d] OUT namespaces/openshift-operators/oc_output/
[must-gather-csc6d] OUT namespaces/openshift-operators/oc_output/configmap
[must-gather-csc6d] OUT namespaces/openshift-operators/oc_output/desc_configmap
[must-gather-csc6d] OUT namespaces/openshift-operators/oc_output/pods
[must-gather-csc6d] OUT namespaces/openshift-operators/oc_output/secrets
[must-gather-csc6d] OUT namespaces/openshift-operators/pods/
[must-gather-csc6d] OUT namespaces/openshift-operators/pods/odf-multicluster-console-6cff9688d5-g254k/
[must-gather-csc6d] OUT namespaces/openshift-operators/pods/odf-multicluster-console-6cff9688d5-g254k/odf-multicluster-console-6cff9688d5-g254k.yaml
[must-gather-csc6d] OUT namespaces/openshift-operators/pods/odf-multicluster-console-6cff9688d5-g254k/odf-multicluster-console/
[must-gather-csc6d] OUT namespaces/openshift-operators/pods/odf-multicluster-console-6cff9688d5-g254k/odf-multicluster-console/odf-multicluster-console/
[must-gather-csc6d] OUT namespaces/openshift-operators/pods/odf-multicluster-console-6cff9688d5-g254k/odf-multicluster-console/odf-multicluster-console/logs/
[must-gather-csc6d] OUT namespaces/openshift-operators/pods/odf-multicluster-console-6cff9688d5-g254k/odf-multicluster-console/odf-multicluster-console/logs/current.log
[must-gather-csc6d] OUT namespaces/openshift-operators/pods/odfmo-controller-manager-56d6b5d96-7sr8h/
[must-gather-csc6d] OUT namespaces/openshift-operators/pods/odfmo-controller-manager-56d6b5d96-7sr8h/odfmo-controller-manager-56d6b5d96-7sr8h.yaml
[must-gather-csc6d] OUT namespaces/openshift-operators/pods/odfmo-controller-manager-56d6b5d96-7sr8h/manager/
[must-gather-csc6d] OUT namespaces/openshift-operators/pods/odfmo-controller-manager-56d6b5d96-7sr8h/manager/manager/
[must-gather-csc6d] OUT namespaces/openshift-operators/pods/odfmo-controller-manager-56d6b5d96-7sr8h/manager/manager/logs/
[must-gather-csc6d] OUT namespaces/openshift-operators/pods/odfmo-controller-manager-56d6b5d96-7sr8h/manager/manager/logs/current.log
[must-gather-csc6d] OUT namespaces/openshift-operators/pods/ramen-hub-operator-687874d5b6-28s8x/
[must-gather-csc6d] OUT namespaces/openshift-operators/pods/ramen-hub-operator-687874d5b6-28s8x/ramen-hub-operator-687874d5b6-28s8x.yaml
[must-gather-csc6d] OUT namespaces/openshift-operators/pods/ramen-hub-operator-687874d5b6-28s8x/kube-rbac-proxy/
[must-gather-csc6d] OUT namespaces/openshift-operators/pods/ramen-hub-operator-687874d5b6-28s8x/kube-rbac-proxy/kube-rbac-proxy/
[must-gather-csc6d] OUT namespaces/openshift-operators/pods/ramen-hub-operator-687874d5b6-28s8x/kube-rbac-proxy/kube-rbac-proxy/logs/
[must-gather-csc6d] OUT namespaces/openshift-operators/pods/ramen-hub-operator-687874d5b6-28s8x/kube-rbac-proxy/kube-rbac-proxy/logs/current.log
[must-gather-csc6d] OUT namespaces/openshift-operators/pods/ramen-hub-operator-687874d5b6-28s8x/manager/
[must-gather-csc6d] OUT namespaces/openshift-operators/pods/ramen-hub-operator-687874d5b6-28s8x/manager/manager/
[must-gather-csc6d] OUT namespaces/openshift-operators/pods/ramen-hub-operator-687874d5b6-28s8x/manager/manager/logs/
[must-gather-csc6d] OUT namespaces/openshift-operators/pods/ramen-hub-operator-687874d5b6-28s8x/manager/manager/logs/current.log
[must-gather-csc6d] OUT namespaces/openshift-operators/pods/ramen-hub-operator-687874d5b6-28s8x/manager/manager/logs/previous.log
[must-gather-csc6d] OUT namespaces/openshift-operators/policy/
[must-gather-csc6d] OUT namespaces/openshift-operators/policy/poddisruptionbudgets.yaml
[must-gather-csc6d] OUT namespaces/openshift-operators/route.openshift.io/
[must-gather-csc6d] OUT namespaces/openshift-operators/route.openshift.io/routes.yaml
[must-gather-csc6d] OUT
[must-gather-csc6d] OUT sent 958 bytes  received 285,146 bytes  572,208.00 bytes/sec
[must-gather-csc6d] OUT total size is 2,941,049  speedup is 10.28
[must-gather      ] OUT namespace/openshift-must-gather-mq5hb deleted
[must-gather      ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-p82hc deleted


Reprinting Cluster State:
When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information:
ClusterID: ed9b1001-7d73-4406-abce-d674408ea07b
ClientVersion: 4.15.0-rc.5
ClusterVersion: Stable at "4.15.0-rc.5"
ClusterOperators:
	All healthy and stable

Comment 3 Aaruni Aggarwal 2024-02-28 08:12:44 UTC

Error is not seen in managed clusters. 

[root@rdr-primary-bastion-0 ~]# oc adm must-gather --image=quay.io/rhceph-dev/ocs-must-gather:latest-4.15 -- /usr/bin/gather --dr
[must-gather      ] OUT Using must-gather plug-in image: quay.io/rhceph-dev/ocs-must-gather:latest-4.15
When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information:
ClusterID: 7f79db65-7458-4c1a-a825-b78bd0dc1a1c
ClientVersion: 4.15.0-rc.5
ClusterVersion: Stable at "4.15.0-rc.5"
ClusterOperators:
	All healthy and stable


[must-gather      ] OUT namespace/openshift-must-gather-6nh6b created
[must-gather      ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-p566v created
[must-gather      ] OUT pod for plug-in image quay.io/rhceph-dev/ocs-must-gather:latest-4.15 created
[must-gather-ddvw2] POD 2024-02-28T05:16:47.550637253Z volume percentage checker started.....
[must-gather-ddvw2] POD 2024-02-28T05:16:47.557218833Z volume usage percentage 0
[must-gather-ddvw2] POD 2024-02-28T05:16:47.562195217Z collection started at: 05:16:47 AM
[must-gather-ddvw2] POD 2024-02-28T05:16:48.653513682Z checking for existing must-gather resource
[must-gather-ddvw2] POD 2024-02-28T05:16:48.753386462Z No resources found in openshift-storage namespace.
[must-gather-ddvw2] POD 2024-02-28T05:16:48.902612208Z creating helper pod
[must-gather-ddvw2] POD 2024-02-28T05:16:50.565505085Z pod/must-gather-ddvw2-helper created
[must-gather-ddvw2] POD 2024-02-28T05:16:50.568686123Z debugging node worker-0
[must-gather-ddvw2] POD 2024-02-28T05:16:50.568850377Z debugging node worker-1
[must-gather-ddvw2] POD 2024-02-28T05:16:50.569051379Z debugging node worker-2
[must-gather-ddvw2] POD 2024-02-28T05:16:50.731061780Z pod/must-gather-ddvw2-helper labeled
[must-gather-ddvw2] POD 2024-02-28T05:16:50.738390524Z waiting for 260 261 262 265 to terminate
[must-gather-ddvw2] POD 2024-02-28T05:16:50.746936676Z Starting pod/worker-2-debug ...
[must-gather-ddvw2] POD 2024-02-28T05:16:50.747057260Z To use host binaries, run `chroot /host`
[must-gather-ddvw2] POD 2024-02-28T05:16:50.751554556Z Starting pod/worker-0-debug ...
[must-gather-ddvw2] POD 2024-02-28T05:16:50.751635187Z To use host binaries, run `chroot /host`
[must-gather-ddvw2] POD 2024-02-28T05:16:50.758691908Z Starting pod/worker-1-debug ...
[must-gather-ddvw2] POD 2024-02-28T05:16:50.758691908Z To use host binaries, run `chroot /host`
[must-gather-ddvw2] POD 2024-02-28T05:16:52.565174574Z volume usage percentage 0
[must-gather-ddvw2] POD 2024-02-28T05:16:57.572245260Z volume usage percentage 0
[must-gather-ddvw2] POD 2024-02-28T05:17:02.579106529Z volume usage percentage 0
[must-gather-ddvw2] POD 2024-02-28T05:17:07.585672299Z volume usage percentage 0
[must-gather-ddvw2] POD 2024-02-28T05:17:12.592313558Z volume usage percentage 0
[must-gather-ddvw2] POD 2024-02-28T05:17:17.599336504Z volume usage percentage 0
[must-gather-ddvw2] POD 2024-02-28T05:17:22.606147999Z volume usage percentage 0
[must-gather-ddvw2] POD 2024-02-28T05:17:27.613350591Z volume usage percentage 0
[must-gather-ddvw2] POD 2024-02-28T05:17:32.620242966Z volume usage percentage 0
[must-gather-ddvw2] POD 2024-02-28T05:17:37.626619464Z volume usage percentage 0
[must-gather-ddvw2] POD 2024-02-28T05:17:42.633759568Z volume usage percentage 0
[must-gather-ddvw2] POD 2024-02-28T05:17:47.641690795Z volume usage percentage 0
[must-gather-ddvw2] POD 2024-02-28T05:17:50.891683092Z pod/must-gather-ddvw2-helper condition met
[must-gather-ddvw2] POD 2024-02-28T05:17:51.128880504Z pod/worker-2-debug condition met
[must-gather-ddvw2] POD 2024-02-28T05:17:51.150407732Z pod/worker-0-debug condition met
[must-gather-ddvw2] POD 2024-02-28T05:17:51.151353751Z pod/worker-1-debug condition met
[must-gather-ddvw2] POD 2024-02-28T05:17:51.540958361Z pod/worker-2-debug labeled
[must-gather-ddvw2] POD 2024-02-28T05:17:51.575193370Z pod/worker-1-debug labeled
[must-gather-ddvw2] POD 2024-02-28T05:17:51.586921159Z pod/worker-0-debug labeled
[must-gather-ddvw2] POD 2024-02-28T05:17:51.589365132Z Collect DR logs...
[must-gather-ddvw2] POD 2024-02-28T05:17:51.589890383Z Waiting on subprocesses to finish execution.
[must-gather-ddvw2] POD 2024-02-28T05:17:51.725736271Z collecting dump of openshift-dr-system namespace
[must-gather-ddvw2] POD 2024-02-28T05:17:52.173854895Z Gathering data for ns/openshift-operators...
[must-gather-ddvw2] POD 2024-02-28T05:17:52.173854895Z W0228 05:17:51.972088     650 warnings.go:70] apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
[must-gather-ddvw2] POD 2024-02-28T05:17:52.173854895Z Wrote inspect data to must-gather.
[must-gather-ddvw2] POD 2024-02-28T05:17:52.177687344Z collecting oc command resource
[must-gather-ddvw2] POD 2024-02-28T05:17:52.268485490Z collecting oc command resource
[must-gather-ddvw2] POD 2024-02-28T05:17:52.361446767Z collecting oc command resource
[must-gather-ddvw2] POD 2024-02-28T05:17:52.446537144Z No resources found in openshift-operators namespace.
[must-gather-ddvw2] POD 2024-02-28T05:17:52.449803108Z collecting oc command resource
[must-gather-ddvw2] POD 2024-02-28T05:17:52.503378741Z You must specify the type of resource to get. Use "oc api-resources" for a complete list of supported resources.
[must-gather-ddvw2] POD 2024-02-28T05:17:52.503378741Z
[must-gather-ddvw2] POD 2024-02-28T05:17:52.503450409Z error: Required resource not specified.
[must-gather-ddvw2] POD 2024-02-28T05:17:52.503450409Z Use "oc explain <resource>" for a detailed description of that resource (e.g. oc explain pods).
[must-gather-ddvw2] POD 2024-02-28T05:17:52.503450409Z See 'oc get -h' for help and examples
[must-gather-ddvw2] POD 2024-02-28T05:17:52.506944457Z collecting oc describe configmap -n openshift-operators
[must-gather-ddvw2] POD 2024-02-28T05:17:52.650089107Z volume usage percentage 0
[must-gather-ddvw2] POD 2024-02-28T05:17:53.672974445Z pod "must-gather-ddvw2-helper" deleted
[must-gather-ddvw2] POD 2024-02-28T05:17:57.656567773Z volume usage percentage 0
[must-gather-ddvw2] POD 2024-02-28T05:18:02.663494939Z volume usage percentage 0
[must-gather-ddvw2] POD 2024-02-28T05:18:07.670222365Z volume usage percentage 0
[must-gather-ddvw2] POD 2024-02-28T05:18:12.677394643Z volume usage percentage 0
[must-gather-ddvw2] POD 2024-02-28T05:18:17.684145573Z volume usage percentage 0
[must-gather-ddvw2] POD 2024-02-28T05:18:22.691320699Z volume usage percentage 0
[must-gather-ddvw2] POD 2024-02-28T05:18:25.256037215Z pod "worker-0-debug" deleted
[must-gather-ddvw2] POD 2024-02-28T05:18:26.313364623Z
[must-gather-ddvw2] POD 2024-02-28T05:18:26.313364623Z Removing debug pod ...
[must-gather-ddvw2] POD 2024-02-28T05:18:26.714181053Z pod "worker-1-debug" deleted
[must-gather-ddvw2] POD 2024-02-28T05:18:27.459427913Z
[must-gather-ddvw2] POD 2024-02-28T05:18:27.459427913Z Removing debug pod ...
[must-gather-ddvw2] POD 2024-02-28T05:18:27.698588952Z volume usage percentage 0
[must-gather-ddvw2] POD 2024-02-28T05:18:27.804141713Z pod "worker-2-debug" deleted
[must-gather-ddvw2] POD 2024-02-28T05:18:28.215970109Z
[must-gather-ddvw2] POD 2024-02-28T05:18:28.215970109Z Removing debug pod ...
[must-gather-ddvw2] OUT waiting for gather to complete
[must-gather-ddvw2] OUT downloading gather output
[must-gather-ddvw2] OUT receiving incremental file list
[must-gather-ddvw2] OUT ./
[must-gather-ddvw2] OUT event-filter.html
[must-gather-ddvw2] OUT gather-debug.log
[must-gather-ddvw2] OUT timestamp
[must-gather-ddvw2] OUT namespaces/
[must-gather-ddvw2] OUT namespaces/openshift-operators/
[must-gather-ddvw2] OUT namespaces/openshift-operators/openshift-operators.yaml
[must-gather-ddvw2] OUT namespaces/openshift-operators/apps.openshift.io/
[must-gather-ddvw2] OUT namespaces/openshift-operators/apps.openshift.io/deploymentconfigs.yaml
[must-gather-ddvw2] OUT namespaces/openshift-operators/apps/
[must-gather-ddvw2] OUT namespaces/openshift-operators/apps/daemonsets.yaml
[must-gather-ddvw2] OUT namespaces/openshift-operators/apps/deployments.yaml
[must-gather-ddvw2] OUT namespaces/openshift-operators/apps/replicasets.yaml
[must-gather-ddvw2] OUT namespaces/openshift-operators/apps/statefulsets.yaml
[must-gather-ddvw2] OUT namespaces/openshift-operators/autoscaling/
[must-gather-ddvw2] OUT namespaces/openshift-operators/autoscaling/horizontalpodautoscalers.yaml
[must-gather-ddvw2] OUT namespaces/openshift-operators/batch/
[must-gather-ddvw2] OUT namespaces/openshift-operators/batch/cronjobs.yaml
[must-gather-ddvw2] OUT namespaces/openshift-operators/batch/jobs.yaml
[must-gather-ddvw2] OUT namespaces/openshift-operators/build.openshift.io/
[must-gather-ddvw2] OUT namespaces/openshift-operators/build.openshift.io/buildconfigs.yaml
[must-gather-ddvw2] OUT namespaces/openshift-operators/build.openshift.io/builds.yaml
[must-gather-ddvw2] OUT namespaces/openshift-operators/core/
[must-gather-ddvw2] OUT namespaces/openshift-operators/core/configmaps.yaml
[must-gather-ddvw2] OUT namespaces/openshift-operators/core/endpoints.yaml
[must-gather-ddvw2] OUT namespaces/openshift-operators/core/events.yaml
[must-gather-ddvw2] OUT namespaces/openshift-operators/core/persistentvolumeclaims.yaml
[must-gather-ddvw2] OUT namespaces/openshift-operators/core/pods.yaml
[must-gather-ddvw2] OUT namespaces/openshift-operators/core/replicationcontrollers.yaml
[must-gather-ddvw2] OUT namespaces/openshift-operators/core/secrets.yaml
[must-gather-ddvw2] OUT namespaces/openshift-operators/core/services.yaml
[must-gather-ddvw2] OUT namespaces/openshift-operators/discovery.k8s.io/
[must-gather-ddvw2] OUT namespaces/openshift-operators/discovery.k8s.io/endpointslices.yaml
[must-gather-ddvw2] OUT namespaces/openshift-operators/image.openshift.io/
[must-gather-ddvw2] OUT namespaces/openshift-operators/image.openshift.io/imagestreams.yaml
[must-gather-ddvw2] OUT namespaces/openshift-operators/k8s.ovn.org/
[must-gather-ddvw2] OUT namespaces/openshift-operators/k8s.ovn.org/egressfirewalls.yaml
[must-gather-ddvw2] OUT namespaces/openshift-operators/networking.k8s.io/
[must-gather-ddvw2] OUT namespaces/openshift-operators/networking.k8s.io/networkpolicies.yaml
[must-gather-ddvw2] OUT namespaces/openshift-operators/oc_output/
[must-gather-ddvw2] OUT namespaces/openshift-operators/oc_output/configmap
[must-gather-ddvw2] OUT namespaces/openshift-operators/oc_output/desc_configmap
[must-gather-ddvw2] OUT namespaces/openshift-operators/oc_output/secrets
[must-gather-ddvw2] OUT namespaces/openshift-operators/policy/
[must-gather-ddvw2] OUT namespaces/openshift-operators/policy/poddisruptionbudgets.yaml
[must-gather-ddvw2] OUT namespaces/openshift-operators/route.openshift.io/
[must-gather-ddvw2] OUT namespaces/openshift-operators/route.openshift.io/routes.yaml
[must-gather-ddvw2] OUT
[must-gather-ddvw2] OUT sent 684 bytes  received 20,794 bytes  42,956.00 bytes/sec
[must-gather-ddvw2] OUT total size is 100,556  speedup is 4.68
[must-gather      ] OUT namespace/openshift-must-gather-6nh6b deleted
[must-gather      ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-p566v deleted


Reprinting Cluster State:
When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information:
ClusterID: 7f79db65-7458-4c1a-a825-b78bd0dc1a1c
ClientVersion: 4.15.0-rc.5
ClusterVersion: Stable at "4.15.0-rc.5"
ClusterOperators:
	All healthy and stable

Comment 4 Aaruni Aggarwal 2024-02-28 08:15:21 UTC

hub cluster: 

ACM and MCO version -> 

[root@rdr-hub-bastion-0 ~]# oc get csv -n open-cluster-management
NAME                                               DISPLAY                                      VERSION             REPLACES                            PHASE
advanced-cluster-management.v2.10.0-79             Advanced Cluster Management for Kubernetes   2.10.0-79                                               Succeeded
odf-multicluster-orchestrator.v4.15.0-149.stable   ODF Multicluster Orchestrator                4.15.0-149.stable                                       Succeeded
odr-hub-operator.v4.15.0-149.stable                Openshift DR Hub Operator                    4.15.0-149.stable                                       Succeeded
openshift-gitops-operator.v1.11.1                  Red Hat OpenShift GitOps                     1.11.1              openshift-gitops-operator.v1.11.0   Succeeded

[root@rdr-hub-bastion-0 ~]# ls -lrt
total 72
drwxr-xr-x. 11 root root 4096 Feb 20 07:51 ocp4-helpernode
drwxr-xr-x.  3 root root 4096 Feb 20 07:54 openstack-upi
drwxr-xr-x.  6 root root 4096 Feb 20 08:36 ocp4-playbooks
drwxr-x---.  2 root root   50 Feb 21 15:12 auth
-rw-r--r--.  1 root root 5074 Feb 21 15:12 pull-secret.txt
drwxr-xr-x.  2 root root   16 Feb 21 15:13 bin
-rw-r--r--.  1 root root  568 Feb 21 15:13 auth.yaml
-rw-r--r--.  1 root root  430 Feb 22 12:42 rhacm-icsp.yaml
-rw-------.  1 root root 4577 Feb 22 12:45 pull_secret.yaml
-rwxr-xr-x.  1 root root  222 Feb 22 12:51 env-acm.sh
drwxr-xr-x. 22 root root 4096 Feb 22 23:58 deploy
-rw-r--r--.  1 root root  474 Feb 24 06:45 odf-icsp.yaml
-rw-r--r--.  1 root root 4835 Feb 24 13:44 cm-clusters-crt.yaml
-rw-r--r--.  1 root root  246 Feb 24 13:51 submariner-broker.yaml
-rw-r--r--.  1 root root 1789 Feb 26 04:35 odf-catalogsource.yaml
-rw-r--r--.  1 root root  489 Feb 27 07:16 rdr-placement.yaml
-rw-r--r--.  1 root root  344 Feb 27 08:53 gitops-cluster.yaml
drwxr-xr-x.  3 root root  120 Feb 27 23:57 must-gather.local.3798704777504983666
drwxr-xr-x.  3 root root  120 Feb 28 00:17 must-gather.local.4295131539163221544

[root@rdr-hub-bastion-0 ~]# cd must-gather.local.4295131539163221544/3416ef7bb1acdd397e8188304fdcdbcfae4581e10fe07d040be5c41f5dbc5a99/namespaces/openshift-operators/

[root@rdr-hub-bastion-0 openshift-operators]# ls
app.k8s.io  apps.openshift.io  batch               core              image.openshift.io  networking.k8s.io  openshift-operators.yaml  policy
apps        autoscaling        build.openshift.io  discovery.k8s.io  k8s.ovn.org         oc_output          pods                      route.openshift.io

[root@rdr-hub-bastion-0 openshift-operators]# ls oc_output/
configmap  desc_configmap  pods  secrets

pods in openshift-operators NS: 

[root@rdr-hub-bastion-0 ~]# oc get pods -n openshift-operators
NAME                                        READY   STATUS    RESTARTS      AGE
odf-multicluster-console-6cff9688d5-g254k   1/1     Running   0             44h
odfmo-controller-manager-56d6b5d96-7sr8h    1/1     Running   0             44h
ramen-hub-operator-687874d5b6-28s8x         2/2     Running   1 (13h ago)   44h

Comment 6 yati padia 2024-02-28 11:11:58 UTC

Hi,

To me, it seems an error that comes while creating the debug pods, which in case of DR resource collection is not required. 
I will check the details and update the bug.
Anyway, I hope the resources are getting collected as required.


regards
Yati

Comment 12 Sunil Kumar Acharya 2024-09-17 17:19:28 UTC

Moving this non-blocker BZ out of ODf-4.17.0 as part of development freeze.