Bug 2054319

Summary:	must-gather \| gather_metallb_logs can't detect metallb pod
Product:	OpenShift Container Platform	Reporter:	Nikita <nkononov>
Component:	oc	Assignee:	Lior Noy <lnoy>
oc sub component:	oc	QA Contact:	Nikita <nkononov>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	medium
Priority:	medium	CC:	aos-bugs, cgoncalves, lnoy, mfojtik
Version:	4.10	Keywords:	Triaged
Target Milestone:	---
Target Release:	4.11.0
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:
Clones:	2054646 (view as bug list)		Environment:
Last Closed:	2022-08-10 10:49:41 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Bug Depends On:
Bug Blocks:	2054646, 2059777

Description Nikita 2022-02-14 16:55:00 UTC

Description of problem:

must-gather doesn't work for metallb. Take a look below:

OCP version:

> oc version
Client Version: 4.10.0-rc.1
Server Version: 4.10.0-rc.1
Kubernetes Version: v1.23.3+b63be7f

Metallb is installed:

> oc get pods -n metallb-system
NAME                                                  READY   STATUS    RESTARTS   AGE
controller-54c59b7475-9btw6                           2/2     Running   0          104s
metallb-operator-controller-manager-9cb6fb86b-k5ccr   1/1     Running   0          143m
speaker-rb9xb                                         6/6     Running   0          104s
speaker-sg5gj                                         6/6     Running   0          104s

 

> oc adm must-gather – /usr/bin/gather_metallb_logs
[must-gather      ] OUT Using must-gather plug-in image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:90979cdd2bba6c5a63b62d3fa2cc3aca15e3e7c684198b40e915be82f4fdd175
When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information.
ClusterID: 081e25ac-9826-48c5-95f8-2e5c99256979
ClusterVersion: Stable at "4.10.0-rc.1"
ClusterOperators:
        clusteroperator/kube-apiserver is degraded because VirtualResourceAdmissionDegraded: Mutating webhook mutation.gatekeeper.sh matches multiple virtual resources: bindings/v1, localresourceaccessreviews.authorization.openshift.io/v1, localsubjectaccessreviews.authorization.k8s.io/v1, localsubjectaccessreviews.authorization.openshift.io/v1, resourceaccessreviews.authorization.openshift.io/v1, selfsubjectaccessreviews.authorization.k8s.io/v1, selfsubjectrulesreviews.authorization.k8s.io/v1, selfsubjectrulesreviews.authorization.openshift.io/v1, subjectaccessreviews.authorization.k8s.io/v1, subjectaccessreviews.authorization.openshift.io/v1, subjectrulesreviews.authorization.openshift.io/v1.
VirtualResourceAdmissionDegraded: Validating webhook validation.gatekeeper.sh matches multiple virtual resources: bindings/v1, localresourceaccessreviews.authorization.openshift.io/v1, localsubjectaccessreviews.authorization.k8s.io/v1, localsubjectaccessreviews.authorization.openshift.io/v1, resourceaccessreviews.authorization.openshift.io/v1, selfsubjectaccessreviews.authorization.k8s.io/v1, selfsubjectrulesreviews.authorization.k8s.io/v1, selfsubjectrulesreviews.authorization.openshift.io/v1, subjectaccessreviews.authorization.k8s.io/v1, subjectaccessreviews.authorization.openshift.io/v1, subjectrulesreviews.authorization.openshift.io/v1.

[must-gather      ] OUT namespace/openshift-must-gather-zjc5m created
[must-gather      ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-655tw created
[must-gather      ] OUT pod for plug-in image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:90979cdd2bba6c5a63b62d3fa2cc3aca15e3e7c684198b40e915be82f4fdd175 created
[must-gather-mg7hd] POD 2022-02-11T19:58:00.172254127Z INFO: MetalLB not detected. Skipping.
[must-gather-mg7hd] OUT waiting for gather to complete
[must-gather-mg7hd] OUT downloading gather output
[must-gather-mg7hd] OUT receiving incremental file list
[must-gather-mg7hd] OUT ./
[must-gather-mg7hd] OUT 
[must-gather-mg7hd] OUT sent 27 bytes  received 41 bytes  15.11 bytes/sec
[must-gather-mg7hd] OUT total size is 0  speedup is 0.00
[must-gather      ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-655tw deleted
[must-gather      ] OUT namespace/openshift-must-gather-zjc5m deleted

When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information.
ClusterID: 081e25ac-9826-48c5-95f8-2e5c99256979
ClusterVersion: Stable at "4.10.0-rc.1"
ClusterOperators:
        clusteroperator/kube-apiserver is degraded because VirtualResourceAdmissionDegraded: Mutating webhook mutation.gatekeeper.sh matches multiple virtual resources: bindings/v1, localresourceaccessreviews.authorization.openshift.io/v1, localsubjectaccessreviews.authorization.k8s.io/v1, localsubjectaccessreviews.authorization.openshift.io/v1, resourceaccessreviews.authorization.openshift.io/v1, selfsubjectaccessreviews.authorization.k8s.io/v1, selfsubjectrulesreviews.authorization.k8s.io/v1, selfsubjectrulesreviews.authorization.openshift.io/v1, subjectaccessreviews.authorization.k8s.io/v1, subjectaccessreviews.authorization.openshift.io/v1, subjectrulesreviews.authorization.openshift.io/v1.
VirtualResourceAdmissionDegraded: Validating webhook validation.gatekeeper.sh matches multiple virtual resources: bindings/v1, localresourceaccessreviews.authorization.openshift.io/v1, localsubjectaccessreviews.authorization.k8s.io/v1, localsubjectaccessreviews.authorization.openshift.io/v1, resourceaccessreviews.authorization.openshift.io/v1, selfsubjectaccessreviews.authorization.k8s.io/v1, selfsubjectrulesreviews.authorization.k8s.io/v1, selfsubjectrulesreviews.authorization.openshift.io/v1, subjectaccessreviews.authorization.k8s.io/v1, subjectaccessreviews.authorization.openshift.io/v1, subjectrulesreviews.authorization.openshift.io/v1.
Version-Release number of selected component (if applicable):


How reproducible:
Run
oc adm must-gather – /usr/bin/gather_metallb_logs


Actual results:
[must-gather-mg7hd] POD 2022-02-11T19:58:00.172254127Z INFO: MetalLB not detected. Skipping.

Expected results:
Relevant logs should be collected.

Comment 1 Carlos Goncalves 2022-02-14 20:02:08 UTC

It seems the metadata.name in Nikita's deployment is "metallb-operator-sub" and is how it's documented in https://docs.openshift.com/container-platform/4.9/networking/metallb/metallb-operator-install.html.
Perhaps instead of filtering by such user-provided/arbitrary field, we could filter by package?

$ oc get subs -A
NAMESPACE                              NAME                                      PACKAGE                      SOURCE                       CHANNEL
local-storage                          local-storage-operator-subscription       local-storage-operator       internal-registry            4.10
metallb-system                         metallb-operator-sub                      metallb-operator             internal-registry            4.10
openshift-performance-addon-operator   performance-addon-operator-subscription   performance-addon-operator   performance-addon-operator   4.10
openshift-ptp                          ptp-operator-subscription                 ptp-operator                 internal-registry            stable
openshift-sriov-network-operator       sriov-network-operator-subscription       sriov-network-operator       internal-registry            4.10

Lior, could you please have a look at this issue? Thanks!

Comment 3 Lior Noy 2022-02-23 08:55:33 UTC

The fix PR got lgtm, waiting for the maintainer's action

Comment 6 Lior Noy 2022-03-02 09:02:57 UTC

(In reply to Lior Noy from comment #3)
> The fix PR got lgtm, waiting for the maintainer's action

Also, note that I verified the solution on 2 scenarios:
1. Install the operator via operator hub.
2. Install via yamls. (following the official metallb Openshift doc)

Comment 7 Scott Dodson 2022-07-13 01:59:36 UTC

Nikita, can you verify this bug please?

Comment 8 Nikita 2022-07-13 18:12:07 UTC

Verified on top of Target Release 4.11
Looks good
> oc version
Server Version: 4.11.0-rc.0
Kubernetes Version: v1.24.0+9ddc8b1

oc adm must-gather -- /usr/bin/gather_metallb
[must-gather      ] OUT Using must-gather plug-in image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e9992ef35b4dc36401e0ea486acdeb4248489b44f4526d75a929815dbfb12c60
When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information.
ClusterID: 0dc7ce73-efd2-4d37-938e-b11c10b16819
ClusterVersion: Stable at "4.11.0-rc.0"
ClusterOperators:
	All healthy and stable


[must-gather      ] OUT namespace/openshift-must-gather-sbjbm created
[must-gather      ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-526k9 created
W0713 14:06:35.682966   46499 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (containers "gather", "copy" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "gather", "copy" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or containers "gather", "copy" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "gather", "copy" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
[must-gather      ] OUT pod for plug-in image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e9992ef35b4dc36401e0ea486acdeb4248489b44f4526d75a929815dbfb12c60 created
[must-gather-zw5nv] POD 2022-07-13T18:06:38.600029837Z Gathering data for ns/metallb-system...
[must-gather-zw5nv] POD 2022-07-13T18:06:39.081881846Z Wrote inspect data to must-gather.
[must-gather-zw5nv] POD 2022-07-13T18:06:39.244968041Z Warning: metallb.io v1beta1 AddressPool is deprecated, consider using IPAddressPool
[must-gather-zw5nv] POD 2022-07-13T18:06:39.247366498Z Wrote inspect data to must-gather.
[must-gather-zw5nv] POD 2022-07-13T18:06:39.415710320Z Wrote inspect data to must-gather.
[must-gather-zw5nv] POD 2022-07-13T18:06:39.592691377Z Wrote inspect data to must-gather.
[must-gather-zw5nv] POD 2022-07-13T18:06:39.788048348Z Wrote inspect data to must-gather.
[must-gather-zw5nv] POD 2022-07-13T18:06:39.915988834Z Wrote inspect data to must-gather.
[must-gather-zw5nv] POD 2022-07-13T18:06:40.021353802Z Wrote inspect data to must-gather.
[must-gather-zw5nv] POD 2022-07-13T18:06:40.109322692Z Wrote inspect data to must-gather.
[must-gather-zw5nv] OUT waiting for gather to complete
[must-gather-zw5nv] OUT downloading gather output
[must-gather-zw5nv] OUT receiving incremental file list
[must-gather-zw5nv] OUT ./
[must-gather-zw5nv] OUT event-filter.html
[must-gather-zw5nv] OUT timestamp
[must-gather-zw5nv] OUT namespaces/
[must-gather-zw5nv] OUT namespaces/metallb-system/
[must-gather-zw5nv] OUT namespaces/metallb-system/metallb-system.yaml
[must-gather-zw5nv] OUT namespaces/metallb-system/apps.openshift.io/
[must-gather-zw5nv] OUT namespaces/metallb-system/apps.openshift.io/deploymentconfigs.yaml
[must-gather-zw5nv] OUT namespaces/metallb-system/apps/
[must-gather-zw5nv] OUT namespaces/metallb-system/apps/daemonsets.yaml
[must-gather-zw5nv] OUT namespaces/metallb-system/apps/deployments.yaml
[must-gather-zw5nv] OUT namespaces/metallb-system/apps/replicasets.yaml
[must-gather-zw5nv] OUT namespaces/metallb-system/apps/statefulsets.yaml
[must-gather-zw5nv] OUT namespaces/metallb-system/autoscaling/
[must-gather-zw5nv] OUT namespaces/metallb-system/autoscaling/horizontalpodautoscalers.yaml
[must-gather-zw5nv] OUT namespaces/metallb-system/batch/
[must-gather-zw5nv] OUT namespaces/metallb-system/batch/cronjobs.yaml
[must-gather-zw5nv] OUT namespaces/metallb-system/batch/jobs.yaml
[must-gather-zw5nv] OUT namespaces/metallb-system/build.openshift.io/
[must-gather-zw5nv] OUT namespaces/metallb-system/build.openshift.io/buildconfigs.yaml
[must-gather-zw5nv] OUT namespaces/metallb-system/build.openshift.io/builds.yaml
[must-gather-zw5nv] OUT namespaces/metallb-system/core/
[must-gather-zw5nv] OUT namespaces/metallb-system/core/configmaps.yaml
[must-gather-zw5nv] OUT namespaces/metallb-system/core/endpoints.yaml
[must-gather-zw5nv] OUT namespaces/metallb-system/core/events.yaml
[must-gather-zw5nv] OUT namespaces/metallb-system/core/persistentvolumeclaims.yaml
[must-gather-zw5nv] OUT namespaces/metallb-system/core/pods.yaml
[must-gather-zw5nv] OUT namespaces/metallb-system/core/replicationcontrollers.yaml
[must-gather-zw5nv] OUT namespaces/metallb-system/core/secrets.yaml
[must-gather-zw5nv] OUT namespaces/metallb-system/core/services.yaml
[must-gather-zw5nv] OUT namespaces/metallb-system/discovery.k8s.io/
[must-gather-zw5nv] OUT namespaces/metallb-system/discovery.k8s.io/endpointslices.yaml
[must-gather-zw5nv] OUT namespaces/metallb-system/image.openshift.io/
[must-gather-zw5nv] OUT namespaces/metallb-system/image.openshift.io/imagestreams.yaml
[must-gather-zw5nv] OUT namespaces/metallb-system/k8s.ovn.org/
[must-gather-zw5nv] OUT namespaces/metallb-system/k8s.ovn.org/egressfirewalls.yaml
[must-gather-zw5nv] OUT namespaces/metallb-system/networking.k8s.io/
[must-gather-zw5nv] OUT namespaces/metallb-system/networking.k8s.io/networkpolicies.yaml
[must-gather-zw5nv] OUT namespaces/metallb-system/pods/
[must-gather-zw5nv] OUT namespaces/metallb-system/pods/metallb-operator-controller-manager-797478c4c6-hr28p/
[must-gather-zw5nv] OUT namespaces/metallb-system/pods/metallb-operator-controller-manager-797478c4c6-hr28p/metallb-operator-controller-manager-797478c4c6-hr28p.yaml
[must-gather-zw5nv] OUT namespaces/metallb-system/pods/metallb-operator-controller-manager-797478c4c6-hr28p/manager/
[must-gather-zw5nv] OUT namespaces/metallb-system/pods/metallb-operator-controller-manager-797478c4c6-hr28p/manager/manager/
[must-gather-zw5nv] OUT namespaces/metallb-system/pods/metallb-operator-controller-manager-797478c4c6-hr28p/manager/manager/logs/
[must-gather-zw5nv] OUT namespaces/metallb-system/pods/metallb-operator-controller-manager-797478c4c6-hr28p/manager/manager/logs/current.log
[must-gather-zw5nv] OUT namespaces/metallb-system/pods/metallb-operator-controller-manager-797478c4c6-hr28p/manager/manager/logs/previous.insecure.log
[must-gather-zw5nv] OUT namespaces/metallb-system/pods/metallb-operator-controller-manager-797478c4c6-hr28p/manager/manager/logs/previous.log
[must-gather-zw5nv] OUT namespaces/metallb-system/pods/metallb-operator-webhook-server-65cf96c67c-tc8s4/
[must-gather-zw5nv] OUT namespaces/metallb-system/pods/metallb-operator-webhook-server-65cf96c67c-tc8s4/metallb-operator-webhook-server-65cf96c67c-tc8s4.yaml
[must-gather-zw5nv] OUT namespaces/metallb-system/pods/metallb-operator-webhook-server-65cf96c67c-tc8s4/webhook-server/
[must-gather-zw5nv] OUT namespaces/metallb-system/pods/metallb-operator-webhook-server-65cf96c67c-tc8s4/webhook-server/webhook-server/
[must-gather-zw5nv] OUT namespaces/metallb-system/pods/metallb-operator-webhook-server-65cf96c67c-tc8s4/webhook-server/webhook-server/logs/
[must-gather-zw5nv] OUT namespaces/metallb-system/pods/metallb-operator-webhook-server-65cf96c67c-tc8s4/webhook-server/webhook-server/logs/current.log
[must-gather-zw5nv] OUT namespaces/metallb-system/pods/metallb-operator-webhook-server-65cf96c67c-tc8s4/webhook-server/webhook-server/logs/previous.insecure.log
[must-gather-zw5nv] OUT namespaces/metallb-system/pods/metallb-operator-webhook-server-65cf96c67c-tc8s4/webhook-server/webhook-server/logs/previous.log
[must-gather-zw5nv] OUT namespaces/metallb-system/policy/
[must-gather-zw5nv] OUT namespaces/metallb-system/policy/poddisruptionbudgets.yaml
[must-gather-zw5nv] OUT namespaces/metallb-system/route.openshift.io/
[must-gather-zw5nv] OUT namespaces/metallb-system/route.openshift.io/routes.yaml
[must-gather-zw5nv] OUT 
[must-gather-zw5nv] OUT sent 796 bytes  received 50,403 bytes  7,876.77 bytes/sec
[must-gather-zw5nv] OUT total size is 326,183  speedup is 6.37
[must-gather      ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-526k9 deleted
[must-gather      ] OUT namespace/openshift-must-gather-sbjbm deleted


When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information.
ClusterID: 0dc7ce73-efd2-4d37-938e-b11c10b16819
ClusterVersion: Stable at "4.11.0-rc.0"
ClusterOperators:
	All healthy and stable

Comment 9 errata-xmlrpc 2022-08-10 10:49:41 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069