Bug 1901675

Summary: [sig-network] multicast when using one of the plugins 'redhat/openshift-ovs-multitenant, redhat/openshift-ovs-networkpolicy' should allow multicast traffic in namespaces where it is enabled
Product: OpenShift Container Platform Reporter: Lokesh Mandvekar <lsm5>
Component: NetworkingAssignee: Aniket Bhat <anbhat>
Networking sub component: openshift-sdn QA Contact: Weibin Liang <weliang>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: medium CC: aconstan, dosmith, weliang
Version: 4.7   
Target Milestone: ---   
Target Release: 4.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1902054 (view as bug list) Environment:
[sig-network] multicast when using one of the plugins 'redhat/openshift-ovs-multitenant, redhat/openshift-ovs-networkpolicy' should allow multicast traffic in namespaces where it is enabled [sig-network] multicast when using one of the plugins 'redhat/openshift-ovs-multitenant, redhat/openshift-ovs-networkpolicy' should block multicast traffic in namespaces where it is disabled
Last Closed: 2021-02-24 15:35:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1816812    
Bug Blocks: 1902054    

Description Lokesh Mandvekar 2020-11-25 19:38:09 UTC
[Filing under multus, because it sounds most relevant, please move if wrong component/subcomponent]

test:
[sig-network] multicast when using one of the plugins 'redhat/openshift-ovs-multitenant, redhat/openshift-ovs-networkpolicy' should allow multicast traffic in namespaces where it is enabled 

is failing frequently in CI, see search results:
https://search.ci.openshift.org/?maxAge=168h&context=1&type=bug%2Bjunit&name=&maxMatches=5&maxBytes=20971520&groupBy=job&search=%5C%5Bsig-network%5C%5D+multicast+when+using+one+of+the+plugins+%27redhat%2Fopenshift-ovs-multitenant%2C+redhat%2Fopenshift-ovs-networkpolicy%27+should+allow+multicast+traffic+in+namespaces+where+it+is+enabled


https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ocp-4.7-e2e-vsphere-upi/1331542098637230080

Seems to fail most often because of:
[sig-instrumentation][sig-builds][Feature:Builds] Prometheus when installed on the cluster should start and expose a secured proxy and verify build metrics [Suite:openshift/conformance/parallel]
[sig-network] multicast when using one of the plugins 'redhat/openshift-ovs-multitenant, redhat/openshift-ovs-networkpolicy' should allow multicast traffic in namespaces where it is enabled [Suite:openshift/conformance/parallel]

Failure log from CI:
: [sig-imageregistry][Feature:ImageLayers] Image layer subresource should return layers from tagged images [Suite:openshift/conformance/parallel] expand_more 	17s
: [sig-cli] Kubectl client Kubectl copy should copy a file from a running Pod [Suite:openshift/conformance/parallel] [Suite:k8s] expand_more 	5m3s
: [sig-cli] oc debug ensure it works with image streams [Suite:openshift/conformance/parallel] expand_more 	1m17s
: [sig-network][Feature:Router] The HAProxy router should serve the correct routes when scoped to a single namespace and label set [Suite:openshift/conformance/parallel] expand_more 	3m27s
: [sig-network][Feature:Router] The HAProxy router should override the route host for overridden domains with a custom value [Suite:openshift/conformance/parallel] expand_more 	3m26s
: [sig-imageregistry][Feature:ImageAppend] Image append should create images by appending them [Suite:openshift/conformance/parallel] expand_more 	24s
: [sig-imageregistry][Feature:Image] oc tag should preserve image reference for external images [Suite:openshift/conformance/parallel] expand_more 	5m18s
: [sig-apps][Feature:Jobs] Users should be able to create and run a job in a user project [Suite:openshift/conformance/parallel] expand_more 	3m18s
: [sig-network][Feature:Router] The HAProxy router should support reencrypt to services backed by a serving certificate automatically [Suite:openshift/conformance/parallel] expand_more 	3m25s
: [sig-builds][Feature:Builds] Optimized image builds should succeed [Suite:openshift/conformance/parallel] expand_more 	2m19s
: [sig-builds][Feature:Builds][valueFrom] process valueFrom in build strategy environment variables should successfully resolve valueFrom in docker build environment variables [Suite:openshift/conformance/parallel] expand_more 	2m35s
: [sig-auth][Feature:SecurityContextConstraints] TestPodDefaultCapabilities [Suite:openshift/conformance/parallel] expand_more 	3m17s
: [sig-cluster-lifecycle] Pods cannot access the /config/master API endpoint [Suite:openshift/conformance/parallel] expand_more 	3m17s
: [sig-apps][Feature:DeploymentConfig] deploymentconfigs when tagging images should successfully tag the deployed image [Suite:openshift/conformance/parallel] expand_more 	5m19s
: [sig-imageregistry][Feature:Image] oc tag should change image reference for internal images [Suite:openshift/conformance/parallel] expand_more 	18s
: [sig-imageregistry][Feature:ImageLookup] Image policy should perform lookup when the Deployment gets the resolve-names annotation later [Suite:openshift/conformance/parallel] expand_more 	17s
: [sig-imageregistry][Feature:Image] oc tag should work when only imagestreams api is available [Suite:openshift/conformance/parallel] expand_more 	5m18s
: [sig-cli] oc debug deployment configs from a build [Suite:openshift/conformance/parallel] expand_more 	5m18s
: [sig-network][Feature:Router] The HAProxy router should expose prometheus metrics for a route [Suite:openshift/conformance/parallel] expand_more 	1m37s
: [sig-auth][Feature:LDAP] LDAP IDP should authenticate against an ldap server [Suite:openshift/conformance/parallel] expand_more 	5m18s
: [sig-imageregistry][Feature:ImageLookup] Image policy should perform lookup when the object has the resolve-names annotation [Suite:openshift/conformance/parallel] expand_more 	18s
: [sig-imageregistry][Feature:ImageExtract] Image extract should extract content from an image [Suite:openshift/conformance/parallel] expand_more 	20s
: [sig-auth][Feature:LDAP] LDAP should start an OpenLDAP test server [Suite:openshift/conformance/parallel] expand_more 	5m18s
: [sig-instrumentation][sig-builds][Feature:Builds] Prometheus when installed on the cluster should start and expose a secured proxy and verify build metrics [Suite:openshift/conformance/parallel] expand_more 	2m46s
: [sig-cli] oc rsh rsh specific flags should work well when access to a remote shell [Suite:openshift/conformance/parallel] expand_more 	4m16s
: [sig-network][Feature:Router] The HAProxy router should run even if it has no access to update status [Suite:openshift/conformance/parallel] expand_more 	3m22s
: [sig-builds][Feature:Builds] Multi-stage image builds should succeed [Suite:openshift/conformance/parallel] expand_more 	2m19s
: [sig-network] Conntrack should be able to preserve UDP traffic when server pod cycles for a NodePort service [Suite:openshift/conformance/parallel] [Suite:k8s] expand_more 	1m20s
: [sig-builds][Feature:Builds] prune builds based on settings in the buildconfig should prune completed builds based on the successfulBuildsHistoryLimit setting [Suite:openshift/conformance/parallel] expand_more 	2m27s
: [sig-imageregistry][Feature:ImageLookup] Image policy should update standard Kube object image fields when local names are on [Suite:openshift/conformance/parallel] expand_more 	17s
: [sig-builds][Feature:Builds] s2i build with a root user image should create a root build and pass with a privileged SCC [Suite:openshift/conformance/parallel] expand_more 	17s
: [sig-network] multicast when using one of the plugins 'redhat/openshift-ovs-multitenant, redhat/openshift-ovs-networkpolicy' should block multicast traffic in namespaces where it is disabled [Suite:openshift/conformance/parallel] expand_more 	5m18s
: [sig-imageregistry][Feature:ImageTriggers] Annotation trigger reconciles after the image is overwritten [Suite:openshift/conformance/parallel] expand_more 	46s
: [sig-network][Feature:Router] The HAProxy router should serve routes that were created from an ingress [Suite:openshift/conformance/parallel] expand_more 	3m56s
: [sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/parallel] expand_more 	3m26s
: [sig-network][Feature:Router] The HAProxy router should serve a route that points to two services and respect weights [Suite:openshift/conformance/parallel] expand_more 	3m22s
: [sig-network] multicast when using one of the plugins 'redhat/openshift-ovs-multitenant, redhat/openshift-ovs-networkpolicy' should allow multicast traffic in namespaces where it is enabled [Suite:openshift/conformance/parallel] expand_more 	5m20s
: [sig-imageregistry][Feature:ImageInfo] Image info should display information about images [Suite:openshift/conformance/parallel] expand_more 	27s
: [sig-builds][Feature:Builds] custom build with buildah being created from new-build should complete build with custom builder image [Suite:openshift/conformance/parallel] expand_more 	3m21s
: [sig-builds][Feature:Builds] build can reference a cluster service with a build being created from new-build should be able to run a build that references a cluster service [Suite:openshift/conformance/parallel] expand_more 	20s
: Run multi-stage test e2e-vsphere-upi - e2e-vsphere-upi-openshift-e2e-test container test expand_more

Comment 1 Lokesh Mandvekar 2020-11-25 19:51:37 UTC
Same log URL for test: [sig-network] multicast when using one of the plugins 'redhat/openshift-ovs-multitenant, redhat/openshift-ovs-networkpolicy' should block multicast traffic in namespaces where it is disabled

Comment 6 Aniket Bhat 2020-12-21 17:11:24 UTC
the azure error seems to be related to some permissions issue:

Dec 14 04:47:19.698: INFO: Error running /usr/bin/oc --namespace=e2e-test-multicast-qvbmb --kubeconfig=/tmp/configfile346677150 exec multicast-3 -- omping -c 5 -T 60 -q -q 10.131.0.188 10.131.0.189 10.128.2.182:
StdOut>
error: You must be logged in to the server (Unauthorized)
StdErr>
error: You must be logged in to the server (Unauthorized)
[AfterEach] [sig-network] multicast
  github.com/openshift/origin/test/extended/util/client.go:138
STEP: Collecting events from namespace "e2e-test-multicast-qvbmb".


The aws error seems to be rhel7 specific and it is likely that the RHEL7 worker node is not set up correctly since the Multus readinessindicatorfile was not found, which likely means the sdn was not properly initialized.

Dec 14 20:00:46.397: INFO: At 2020-12-14 19:57:10 +0000 UTC - event for multicast-3: {kubelet ip-10-0-179-21.us-west-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_multicast-3_e2e-test-multicast-xdhf8_e32d5fb1-150a-4094-b18f-2a9f59c861ab_0(ce76b14dd92b45a4f69ba2402bcc4580c76cc50f58839fdfe58dc75f6e08ec6c): Multus: [e2e-test-multicast-xdhf8/multicast-3]: PollImmediate error waiting for ReadinessIndicatorFile: timed out waiting for the condition
Dec 14 20:00:46.397: INFO: At 2020-12-14 19:58:53 +0000 UTC - event for multicast-3: {kubelet ip-10-0-179-21.us-west-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_multicast-3_e2e-test-multicast-xdhf8_e32d5fb1-150a-4094-b18f-2a9f59c861ab_0(b352d3d09db7f57423b3a313955969f8d3e36c59a3f3a7c9556baf2b56e36b6e): Multus: [e2e-test-multicast-xdhf8/multicast-3]: PollImmediate error waiting for ReadinessIndicatorFile: timed out waiting for the condition
Dec 14 20:00:46.397: INFO: At 2020-12-14 20:00:35 +0000 UTC - event for multicast-3: {kubelet ip-10-0-179-21.us-west-1.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_multicast-3_e2e-test-multicast-xdhf8_e32d5fb1-150a-4094-b18f-2a9f59c861ab_0(18fb08e899765e33d85313fcddb94cc2a7e89e692764e5985e85fdc6c1c806ea): Multus: [e2e-test-multicast-xdhf8/multicast-3]: PollImmediate error waiting for ReadinessIndicatorFile: timed out waiting for the condition

The rhel7 worker issue seems to be related to: https://bugzilla.redhat.com/show_bug.cgi?id=1908616

Comment 8 Weibin Liang 2021-01-04 18:18:44 UTC
Not see multicast failed log in e2e-azure-4.7 any more.

Comment 11 errata-xmlrpc 2021-02-24 15:35:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633