Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1866414

Summary: corosync with two-link knet fails after "ifdown eth1"
Product: Red Hat Enterprise Linux 8 Reporter: Christine Caulfield <ccaulfie>
Component: kronosnetAssignee: Christine Caulfield <ccaulfie>
Status: CLOSED ERRATA QA Contact: cluster-qe <cluster-qe>
Severity: high Docs Contact:
Priority: high    
Version: 8.2CC: cfeist, fdinitto, jfriesse, phagara
Target Milestone: rcKeywords: ZStream
Target Release: 8.0Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1867708 (view as bug list) Environment:
Last Closed: 2020-11-04 02:19:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1867708    

Description Christine Caulfield 2020-08-05 14:14:35 UTC 
From https://bugs.centos.org/view.php?id=17645

Have two nodes conneced to the 172.26.1.x/24 network with their eth0 interfaces, and connected to the 10.0.0.x/24 network with their eth1 interfaces.

Create a pcs cluster:
pcs1# pcs cluster setup yklastr \
        pcs1.yenya.net addr=172.26.1.147 addr=10.0.0.147 \
        pcs2.yenya.net addr=172.26.1.208 addr=10.0.0.208

Wait for it to become online:
pcs1# pcs cluster status

Bring down one interface:
pcs2# ifdown eth1

Expected behaviour: the cluster should communicate on the eth0 network.

Current behaviour: corosync on pcs2 node eats 95-100 % CPU, and the cluster communication does not work (no traffic on eth0).
Additional Information	Rebuilding kronosnet-1.15-1.fc32.src.rpm from Fedora 32, installing the RPMs, and restarting corosync fixes the problem. CentOS 8 RPMs from the upstream repository linked from the github issue above also work.


The fix is upstream as commit b84b48c61368544d14324d6173d3918d24df9837 (I haven't checked dependencies, but they are probably small if anything).

This might be a good candidate for Zstream
Comment 3 Chris Feist 2020-08-10 15:05:35 UTC 
This is the 8.3 version of this bz which is already fixed by the rebase done for 8.3.
Comment 4 Christine Caulfield 2020-08-12 08:55:48 UTC 
Fixed in knet 1.15
Comment 7 Patrik Hagara 2020-09-16 11:41:05 UTC 
before fix (kronosnet-1.10-4.el8)
=================================

> [root@virt-493 ~]# rpm -q libknet1
> libknet1-1.10-4.el8.x86_64
> [root@virt-493 ~]# corosync-cfgtool -s
> Printing link status.
> Local node ID 1
> LINK ID 0
> 	addr	= 10.37.165.141 [ens3]
> 	status:
> 		nodeid  1:	link enabled:1	link connected:1
> 		nodeid  2:	link enabled:1	link connected:1
> LINK ID 1
> 	addr	= 192.168.2.13 [ens5]
> 	status:
> 		nodeid  1:	link enabled:0	link connected:1
> 		nodeid  2:	link enabled:1	link connected:1


take down the ens5 network interface (ring1):

> [root@virt-493 ~]# ifdown ens5
> Connection 'System ens5' successfully deactivated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/2)
> [root@virt-493 ~]# top
>  62789 root      rt   0  734856 254140  89660 S 100.7   6.1   0:09.77 corosync                                                         
> [root@virt-493 ~]# client_loop: send disconnect: Broken pipe

> [root@virt-497 ~]# pcs stonith history
> reboot of virt-493 successful: delegate=virt-497, client=pacemaker-controld.62788, origin=virt-497, completed='2020-09-16 13:37:05 +02:00'
> 1 event found

result: corosync process consumes 100% cpu on the node where network interface was brought down. cluster fails to continue communication, fencing is triggered.


after fix (kronosnet-1.16-1.el8)
================================

> [root@virt-242 ~]# rpm -q libknet1
> libknet1-1.16-1.el8.x86_64
> [root@virt-242 ~]# corosync-cfgtool -s
> Printing link status.
> Local node ID 1
> LINK ID 0
> 	addr	= 10.37.167.113 [ens3]
> 	status:
> 		nodeid  1:	link enabled:1	link connected:1
> 		nodeid  2:	link enabled:1	link connected:1
> LINK ID 1
> 	addr	= 192.168.2.2 [ens5]
> 	status:
> 		nodeid  1:	link enabled:0	link connected:1
> 		nodeid  2:	link enabled:1	link connected:1


take down the ens5 network interface (ring1):

> [root@virt-242 ~]# ifdown ens5
> Connection 'System ens5' successfully deactivated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/2)
> [root@virt-242 ~]# corosync-cfgtool -s
> Printing link status.
> Local node ID 1
> LINK ID 0
> 	addr	= 10.37.167.113
> 	status:
> 		nodeid  1:	link enabled:1	link connected:1
> 		nodeid  2:	link enabled:1	link connected:1
> LINK ID 1
> 	addr	= 192.168.2.2
> 	status:
> 		nodeid  1:	link enabled:0	link connected:1
> 		nodeid  2:	link enabled:1	link connected:0

cluster communication continues over ring0 link, cpu usage remains negligible.


reactivate the ring1 link:

> [root@virt-242 ~]# ifup ens5
> Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/4)
> [root@virt-242 ~]# corosync-cfgtool -s
> Printing link status.
> Local node ID 1
> LINK ID 0
> 	addr	= 10.37.167.113
> 	status:
> 		nodeid  1:	link enabled:1	link connected:1
> 		nodeid  2:	link enabled:1	link connected:1
> LINK ID 1
> 	addr	= 192.168.2.2
> 	status:
> 		nodeid  1:	link enabled:0	link connected:1
> 		nodeid  2:	link enabled:1	link connected:1

ring1 link is successfully connected again, cluster continues operating, no abnormal cpu load.


marking verified in kronosnet-1.16-1.el8.
Comment 10 errata-xmlrpc 2020-11-04 02:19:48 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (kronosnet bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4606