Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1896493

Summary:	log spam after runtime reconfiguration of crypto to use HMAC only without encryption
Product:	Red Hat Enterprise Linux 8	Reporter:	Patrik Hagara <phagara>
Component:	corosync	Assignee:	Jan Friesse <jfriesse>
Status:	CLOSED ERRATA	QA Contact:	cluster-qe <cluster-qe>
Severity:	low	Docs Contact:
Priority:	unspecified
Version:	8.4	CC:	ccaulfie, cluster-maint, sfoucek
Target Milestone:	rc	Flags:	pm-rhel: mirror+
Target Release:	8.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	corosync-3.1.0-3.el8	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-05-18 15:26:09 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Patrik Hagara 2020-11-10 17:39:55 UTC

Description of problem:

While testing bz#1855303 feature, it was discovered that using HMAC packet authentication only (without encryption) spams the corosync logs with digest and packet size mismatch errors. Strangely enough, the clusters seems to continue operating just fine.

Configure a cluster using pcs, leaving default crypto settings (aes256/sha256):

> [root@virt-248 ~]# rpm -q pacemaker corosync libknet1
> pacemaker-2.0.5-2.el8.x86_64
> corosync-3.1.0-1.el8.x86_64
> libknet1-1.18-1.el8.x86_64
> [root@virt-248 ~]# corosync-cfgtool -s
> Printing link status.
> Local node ID 1
> LINK ID 0
>         addr    = 2620:52:0:25a4:1800:ff:fe00:f8
>         status:
>                 nodeid  1:      localhost
>                 nodeid  2:      connected
>                 nodeid  3:      connected
>                 nodeid  4:      connected
>                 nodeid  5:      connected
> [root@virt-248 ~]# corosync-cmapctl totem.crypto_cipher totem.crypto_hash
> totem.crypto_cipher (str) = aes256
> totem.crypto_hash (str) = sha256
> [root@virt-248 ~]# OTHER_NODES="virt-249 virt-250 virt-251 virt-252"


Using the new live crypto reconfiguration feature from bz#1855303, change token.crypto_cipher to "none" and issue the reload command:

> [root@virt-248 ~]# sed -Ei 's/(crypto_cipher: ).*/\1none/' /etc/corosync/corosync.conf
> [root@virt-248 ~]# for node in $OTHER_NODES; do scp /etc/corosync/corosync.conf $node:/etc/corosync/; done
> corosync.conf                                                                                         100%  735     1.0MB/s   00:00
> corosync.conf                                                                                         100%  735     1.1MB/s   00:00
> corosync.conf                                                                                         100%  735     1.4MB/s   00:00
> corosync.conf                                                                                         100%  735     1.3MB/s   00:00
> [root@virt-248 ~]# corosync-cfgtool -R
> Reloading corosync.conf...
> Done
> [root@virt-248 ~]# for node in $OTHER_NODES; do echo $node; ssh $node corosync-cmapctl totem.crypto_cipher totem.crypto_hash; corosync-cfgtool -s; done
> virt-249
> totem.crypto_cipher (str) = none
> totem.crypto_hash (str) = sha256
> Printing link status.
> Local node ID 1
> LINK ID 0
>         addr    = 2620:52:0:25a4:1800:ff:fe00:f8
>         status:
>                 nodeid  1:      localhost
>                 nodeid  2:      connected
>                 nodeid  3:      connected
>                 nodeid  4:      connected
>                 nodeid  5:      connected
> virt-250
> totem.crypto_cipher (str) = none
> totem.crypto_hash (str) = sha256
> Printing link status.
> Local node ID 1
> LINK ID 0
>         addr    = 2620:52:0:25a4:1800:ff:fe00:f8
>         status:
>                 nodeid  1:      localhost
>                 nodeid  2:      connected
>                 nodeid  3:      connected
>                 nodeid  4:      connected
>                 nodeid  5:      connected
> virt-251
> totem.crypto_cipher (str) = none
> totem.crypto_hash (str) = sha256
> Printing link status.
> Local node ID 1
> LINK ID 0
>         addr    = 2620:52:0:25a4:1800:ff:fe00:f8
>         status:
>                 nodeid  1:      localhost
>                 nodeid  2:      connected
>                 nodeid  3:      connected
>                 nodeid  4:      connected
>                 nodeid  5:      connected
> virt-252
> totem.crypto_cipher (str) = none
> totem.crypto_hash (str) = sha256
> Printing link status.
> Local node ID 1
> LINK ID 0
>         addr    = 2620:52:0:25a4:1800:ff:fe00:f8
>         status:
>                 nodeid  1:      localhost
>                 nodeid  2:      connected
>                 nodeid  3:      connected
>                 nodeid  4:      connected
>                 nodeid  5:      connected


The reconfiguration seems to succeed. However, all nodes keep spamming the corosync.log with crypto and packet size errors:

> [root@virt-248 ~]# tail -f /var/log/cluster/corosync.log
> Nov 09 10:58:23 [50162] virt-248.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Digest does not match
> Nov 09 10:58:24 [50162] virt-248.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Incorrect packet size.
> Nov 09 10:58:24 [50162] virt-248.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Incorrect packet size.
> Nov 09 10:58:24 [50162] virt-248.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Incorrect packet size.
> Nov 09 10:58:24 [50162] virt-248.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Incorrect packet size.
> Nov 09 10:58:24 [50162] virt-248.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Incorrect packet size.
> Nov 09 10:58:24 [50162] virt-248.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Incorrect packet size.
> Nov 09 10:58:24 [50162] virt-248.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Digest does not match
> Nov 09 10:58:24 [50162] virt-248.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Incorrect packet size.
> Nov 09 10:58:24 [50162] virt-248.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Incorrect packet size.
> Nov 09 10:58:25 [50162] virt-248.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Digest does not match


No cluster membership changes were observed, which seems weird.

Changing the token.crypto_cipher back to "aes256" brings the cluster back to a sane state. Again, no fencing occurs.

> [root@virt-248 ~]# sed -Ei 's/(crypto_cipher: ).*/\1aes256/' /etc/corosync/corosync.conf
> [root@virt-248 ~]# for node in $OTHER_NODES; do scp /etc/corosync/corosync.conf $node:/etc/corosync/; done
> corosync.conf              100%  737   612.0KB/s   00:00
> corosync.conf              100%  737   605.1KB/s   00:00
> corosync.conf              100%  737   553.7KB/s   00:00
> corosync.conf              100%  737   725.3KB/s   00:00
> [root@virt-248 ~]# corosync-cfgtool -R
> Reloading corosync.conf...
> Done
> [root@virt-248 ~]# tail -f /var/log/cluster/corosync.log
> Nov 09 11:03:05 [50162] virt-248.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Digest does not match
> Nov 09 11:03:05 [50162] virt-248.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Digest does not match
> Nov 09 11:03:05 [50162] virt-248.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Digest does not match
> Nov 09 11:03:05 [50162] virt-248.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Digest does not match
> Nov 09 11:03:05 [50162] virt-248.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Digest does not match
> Nov 09 11:03:05 [50162] virt-248.cluster-qe.lab.eng.brq.redhat.com corosync info    [KNET  ] pmtud: PMTUD link change for host: 5 link: 0 from 1426 to 1365
> Nov 09 11:03:05 [50162] virt-248.cluster-qe.lab.eng.brq.redhat.com corosync info    [KNET  ] pmtud: PMTUD link change for host: 4 link: 0 from 1426 to 1365
> Nov 09 11:03:05 [50162] virt-248.cluster-qe.lab.eng.brq.redhat.com corosync info    [KNET  ] pmtud: PMTUD link change for host: 3 link: 0 from 1426 to 1365
> Nov 09 11:03:05 [50162] virt-248.cluster-qe.lab.eng.brq.redhat.com corosync info    [KNET  ] pmtud: PMTUD link change for host: 2 link: 0 from 1426 to 1365
> Nov 09 11:03:05 [50162] virt-248.cluster-qe.lab.eng.brq.redhat.com corosync info    [KNET  ] pmtud: Global data MTU changed to: 1365
[log goes quiet]

The corosync.conf manual page clearly states that setting totem.crypto_cipher to value other than "none" also requires enabling totem.crypto_hash. This is also enforced at configuration (re-)load time with "Could not reload configuration. Error CS_ERR_INVALID_PARAM" error thrown.

However, here the configuration was the other way around -- only crypto_hash enabled while crypto_cipher remained at "none", ie. message authentication only, without encryption (which seems like a valid configuration for a limited set of use-cases).


Version-Release number of selected component (if applicable):
corosync-3.1.0-1.el8.x86_64

How reproducible:
always

Steps to Reproduce:
1. have a running corosync cluster with both totem.crypto_cipher and totem.crypto_hash set to values other than "none" (eg. the default aes256/sha256)
2. change totem.crypto_cipher to "none" in corosync.conf on all nodes
3. issue cororsync-cfgtool -R
4. watch corosync logs

Actual results:
log gets spammed with invalid packet error

Expected results:
no packet errors in the log, cluster communication works with authentication without encryption

Additional info:

Comment 1 Jan Friesse 2020-11-12 12:50:30 UTC

Upstream patch merged as a https://github.com/corosync/corosync/commit/4a2f48b17b06638d3d3adcae683aff1639351434

Comment 4 Simon Foucek 2020-12-02 21:49:06 UTC

Before fix:
>[root@virt-489 ~]# rpm -q pacemaker corosync libknet1
>pacemaker-2.0.5-2.el8.x86_64
>corosync-3.1.0-1.el8.x86_64
>libknet1-1.18-1.el8.x86_64
>[root@virt-489 ~]# corosync-cfgtool -s
>Printing link status.
>Local node ID 1
>LINK ID 0
>	addr	= 2620:52:0:25a4:1800:ff:fe00:1e9
>	status:
>		nodeid  1:	localhost
>		nodeid  2:	connected
>		nodeid  3:	connected
>		nodeid  4:	connected
>		nodeid  5:	connected
>		nodeid  6:	connected
>
>[root@virt-489 ~]# corosync-cmapctl totem.crypto_cipher totem.crypto_hash
>totem.crypto_cipher (str) = aes256
>totem.crypto_hash (str) = sha256
>[root@virt-489 ~]# NODES="virt-490 virt-491 virt-492 virt-493 virt-494"
>[root@virt-489 ~]# sed -Ei 's/(crypto_cipher: ).*/\1none/' /etc/corosync/corosync.conf
>[root@virt-489 ~]# for node in $NODES; do scp /etc/corosync/corosync.conf $node:/etc/corosync/; done
>Warning: Permanently added 'virt-490,2620:52:0:25a4:1800:ff:fe00:1ea' (ECDSA) to the list of known hosts.
>corosync.conf                                                                                                                                                                    100%  824     1.2MB/s   00:00    
>Warning: Permanently added 'virt-491,2620:52:0:25a4:1800:ff:fe00:1eb' (ECDSA) to the list of known hosts.
>corosync.conf                                                                                                                                                                    100%  824     1.2MB/s   00:00    
>Warning: Permanently added 'virt-492,2620:52:0:25a4:1800:ff:fe00:1ec' (ECDSA) to the list of known hosts.
>corosync.conf                                                                                                                                                                    100%  824   996.8KB/s   00:00    
>Warning: Permanently added 'virt-493,2620:52:0:25a4:1800:ff:fe00:1ed' (ECDSA) to the list of known hosts.
>corosync.conf                                                                                                                                                                    100%  824     1.1MB/s   00:00    
>Warning: Permanently added 'virt-494,2620:52:0:25a4:1800:ff:fe00:1ee' (ECDSA) to the list of known hosts.
>corosync.conf                                                                                                                                                                    100%  824     1.2MB/s   00:00    
>[root@virt-489 ~]# corosync-cfgtool -R
>Reloading corosync.conf...
>Done
>[root@virt-489 ~]# for node in $NODES; do echo $node; ssh $node corosync-cmapctl totem.crypto_cipher totem.crypto_hash; corosync-cfgtool -s; done
>virt-490
>totem.crypto_cipher (str) = none
>totem.crypto_hash (str) = sha256
>Printing link status.
>Local node ID 1
>LINK ID 0
>	addr	= 2620:52:0:25a4:1800:ff:fe00:1e9
>	status:
>		nodeid  1:	localhost
>		nodeid  2:	connected
>		nodeid  3:	connected
>		nodeid  4:	connected
>		nodeid  5:	connected
>		nodeid  6:	connected
>virt-491
>totem.crypto_cipher (str) = none
>totem.crypto_hash (str) = sha256
>Printing link status.
>Local node ID 1
>LINK ID 0
>	addr	= 2620:52:0:25a4:1800:ff:fe00:1e9
>	status:
>		nodeid  1:	localhost
>		nodeid  2:	connected
>		nodeid  3:	connected
>		nodeid  4:	connected
>		nodeid  5:	connected
>		nodeid  6:	connected
>virt-492
>totem.crypto_cipher (str) = none
>totem.crypto_hash (str) = sha256
>Printing link status.
>Local node ID 1
>LINK ID 0
>	addr	= 2620:52:0:25a4:1800:ff:fe00:1e9
>	status:
>		nodeid  1:	localhost
>		nodeid  2:	connected
>		nodeid  3:	connected
>		nodeid  4:	connected
>		nodeid  5:	connected
>		nodeid  6:	connected
>virt-493
>totem.crypto_cipher (str) = none
>totem.crypto_hash (str) = sha256
>Printing link status.
>Local node ID 1
>LINK ID 0
>	addr	= 2620:52:0:25a4:1800:ff:fe00:1e9
>	status:
>		nodeid  1:	localhost
>		nodeid  2:	connected
>		nodeid  3:	connected
>		nodeid  4:	connected
>		nodeid  5:	connected
>		nodeid  6:	connected
>virt-494
>totem.crypto_cipher (str) = none
>totem.crypto_hash (str) = sha256
>Printing link status.
>Local node ID 1
>LINK ID 0
>	addr	= 2620:52:0:25a4:1800:ff:fe00:1e9
>	status:
>		nodeid  1:	localhost
>		nodeid  2:	connected
>		nodeid  3:	connected
>		nodeid  4:	connected
>		nodeid  5:	connected
>		nodeid  6:	connected
>[root@virt-489 ~]# tail -f /var/log/cluster/corosync.log
>Dec 02 18:54:37 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Incorrect packet size.
>Dec 02 18:54:37 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Incorrect packet size.
>Dec 02 18:54:37 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Incorrect packet size.
>Dec 02 18:54:37 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Incorrect packet size.
>Dec 02 18:54:37 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Incorrect packet size.
>Dec 02 18:54:37 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Incorrect packet size.
>Dec 02 18:54:37 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Incorrect packet size.
>Dec 02 18:54:37 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Incorrect packet size.
>Dec 02 18:54:37 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Digest does not match
>Dec 02 18:54:37 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Incorrect packet size.
>Dec 02 18:54:38 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Digest does not match
>Dec 02 18:54:38 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Incorrect packet size.
>Dec 02 18:54:39 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Incorrect packet size.
>Dec 02 18:54:39 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Incorrect packet size.
>Dec 02 18:54:39 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Incorrect packet size.
>Dec 02 18:54:39 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Incorrect packet size.
>Dec 02 18:54:39 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Incorrect packet size.
>Dec 02 18:54:39 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Incorrect packet size.
>Dec 02 18:54:39 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Incorrect packet size.
>Dec 02 18:54:39 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Incorrect packet size.
>Dec 02 18:54:39 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Incorrect packet size.
>Dec 02 18:54:39 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Digest does not match
>Dec 02 18:54:40 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Incorrect packet size.
>Dec 02 18:54:40 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Incorrect packet size.
>Dec 02 18:54:40 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Incorrect packet size.
>Dec 02 18:54:40 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Incorrect packet size.
>Dec 02 18:54:40 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Incorrect packet size.
>Dec 02 18:54:40 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Incorrect packet size.
>Dec 02 18:54:40 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Incorrect packet size.
>Dec 02 18:54:40 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Incorrect packet size.
>Dec 02 18:54:40 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Incorrect packet size.
>Dec 02 18:54:40 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Incorrect packet size.
>Dec 02 18:54:40 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Digest does not match
>
>[root@virt-489 ~]# sed -Ei 's/(crypto_cipher: ).*/\1aes256/' /etc/corosync/corosync.conf
>[root@virt-489 ~]# for node in $NODES; do scp /etc/corosync/corosync.conf $node:/etc/corosync/; done
>corosync.conf                                                                                                                                                                    100%  826   305.3KB/s   00:00    
>corosync.conf                                                                                                                                                                    100%  826   897.6KB/s   00:00    
>corosync.conf                                                                                                                                                                    100%  826   733.5KB/s   00:00    
>corosync.conf                                                                                                                                                                    100%  826   785.2KB/s   00:00    
>corosync.conf                                                                                                                                                                    100%  826   718.1KB/s   00:00    
>[root@virt-489 ~]# corosync-cfgtool -R
>Reloading corosync.conf...
>Done
>[root@virt-489 ~]# tail -f /var/log/cluster/corosync.log
>Dec 02 18:55:24 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Digest does not match
>Dec 02 18:55:24 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Digest does not match
>Dec 02 18:55:24 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Digest does not match
>Dec 02 18:55:24 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync error   [KNET  ] nsscrypto: Digest does not match
>Dec 02 18:55:24 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync info    [KNET  ] pmtud: PMTUD link change for host: 6 link: 0 from 1426 to 1365
>Dec 02 18:55:24 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync info    [KNET  ] pmtud: PMTUD link change for host: 5 link: 0 from 1426 to 1365
>Dec 02 18:55:24 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync info    [KNET  ] pmtud: PMTUD link change for host: 4 link: 0 from 1426 to 1365
>Dec 02 18:55:24 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync info    [KNET  ] pmtud: PMTUD link change for host: 3 link: 0 from 1426 to 1365
>Dec 02 18:55:24 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync info    [KNET  ] pmtud: PMTUD link change for host: 2 link: 0 from 1426 to 1365
>Dec 02 18:55:24 [49873] virt-489.cluster-qe.lab.eng.brq.redhat.com corosync info    [KNET  ] pmtud: Global data MTU changed to: 1365

Result: Same as in the first example. After change of token.crypto_cipher to "none" and issue the reload command, all nodes keep spamming the corosync.log with crypto and packet size errors.Changing the token.crypto_cipher back to "aes256" brings the cluster back to a sane state.

After fix:

>[root@virt-038 ~]# rpm -q pacemaker corosync libknet1
>pacemaker-2.0.5-2.el8.x86_64
>corosync-3.1.0-3.el8.x86_64
>libknet1-1.18-1.el8.x86_64
>[root@virt-038 ~]# corosync-cfgtool -s
>Printing link status.
>Local node ID 1
>LINK ID 0
>	addr	= 2620:52:0:25a4:1800:ff:fe00:26
>	status:
>		nodeid  1:	localhost
>		nodeid  2:	connected
>		nodeid  3:	connected
>		nodeid  4:	connected
>		nodeid  5:	connected
>		nodeid  6:	connected
>[root@virt-038 ~]# corosync-cmapctl totem.crypto_cipher totem.crypto_hash
>totem.crypto_cipher (str) = aes256
>totem.crypto_hash (str) = sha256
>[root@virt-038 ~]# NODES="virt-046 virt-047 virt-048 virt-049 virt-050"
>[root@virt-038 ~]# sed -Ei 's/(crypto_cipher: ).*/\1none/' /etc/corosync/corosync.conf
>[root@virt-038 ~]# for node in $NODES; do scp /etc/corosync/corosync.conf $node:/etc/corosync/; done
>Warning: Permanently added 'virt-046,2620:52:0:25a4:1800:ff:fe00:2e' (ECDSA) to the list of known hosts.
>corosync.conf                                                                                                                                                                    100%  824   960.2KB/s   00:00    
>Warning: Permanently added 'virt-047,2620:52:0:25a4:1800:ff:fe00:2f' (ECDSA) to the list of known hosts.
>corosync.conf                                                                                                                                                                    100%  824   959.4KB/s   00:00    
>Warning: Permanently added 'virt-048,2620:52:0:25a4:1800:ff:fe00:30' (ECDSA) to the list of known hosts.
>corosync.conf                                                                                                                                                                    100%  824     1.1MB/s   00:00    
>Warning: Permanently added 'virt-049,2620:52:0:25a4:1800:ff:fe00:31' (ECDSA) to the list of known hosts.
>corosync.conf                                                                                                                                                                    100%  824     1.0MB/s   00:00    
>Warning: Permanently added 'virt-050,2620:52:0:25a4:1800:ff:fe00:32' (ECDSA) to the list of known hosts.
>corosync.conf                                                                                                                                                                    100%  824   954.2KB/s   00:00    
>[root@virt-038 ~]#  corosync-cfgtool -R
>Reloading corosync.conf...
>Done
>[root@virt-038 ~]# for node in $NODES; do echo $node; ssh $node corosync-cmapctl totem.crypto_cipher totem.crypto_hash; corosync-cfgtool -s; done
>virt-046
>totem.crypto_cipher (str) = none
>totem.crypto_hash (str) = sha256
>Printing link status.
>Local node ID 1
>LINK ID 0
>	addr	= 2620:52:0:25a4:1800:ff:fe00:26
>	status:
>		nodeid  1:	localhost
>		nodeid  2:	connected
>		nodeid  3:	connected
>		nodeid  4:	connected
>		nodeid  5:	connected
>		nodeid  6:	connected
>virt-047
>totem.crypto_cipher (str) = none
>totem.crypto_hash (str) = sha256
>Printing link status.
>Local node ID 1
>LINK ID 0
>	addr	= 2620:52:0:25a4:1800:ff:fe00:26
>	status:
>		nodeid  1:	localhost
>		nodeid  2:	connected
>		nodeid  3:	connected
>		nodeid  4:	connected
>		nodeid  5:	connected
>		nodeid  6:	connected
>virt-048
>totem.crypto_cipher (str) = none
>totem.crypto_hash (str) = sha256
>Printing link status.
>Local node ID 1
>LINK ID 0
>	addr	= 2620:52:0:25a4:1800:ff:fe00:26
>	status:
>		nodeid  1:	localhost
>		nodeid  2:	connected
>		nodeid  3:	connected
>		nodeid  4:	connected
>		nodeid  5:	connected
>		nodeid  6:	connected
>virt-049
>totem.crypto_cipher (str) = none
>totem.crypto_hash (str) = sha256
>Printing link status.
>Local node ID 1
>LINK ID 0
>	addr	= 2620:52:0:25a4:1800:ff:fe00:26
>	status:
>		nodeid  1:	localhost
>		nodeid  2:	connected
>		nodeid  3:	connected
>		nodeid  4:	connected
>		nodeid  5:	connected
>		nodeid  6:	connected
>virt-050
>totem.crypto_cipher (str) = none
>totem.crypto_hash (str) = sha256
>Printing link status.
>Local node ID 1
>LINK ID 0
>	addr	= 2620:52:0:25a4:1800:ff:fe00:26
>	status:
>		nodeid  1:	localhost
>		nodeid  2:	connected
>		nodeid  3:	connected
>		nodeid  4:	connected
>		nodeid  5:	connected
>		nodeid  6:	connected
>[root@virt-038 ~]# tail -f /var/log/cluster/corosync.log
>Dec 02 18:23:38 [49066] virt-038.cluster-qe.lab.eng.brq.redhat.com corosync notice  [QUORUM] This node is within the primary component and will provide service.
>Dec 02 18:23:38 [49066] virt-038.cluster-qe.lab.eng.brq.redhat.com corosync notice  [QUORUM] Members[6]: 1 2 3 4 5 6
>Dec 02 18:23:38 [49066] virt-038.cluster-qe.lab.eng.brq.redhat.com corosync notice  [MAIN  ] Completed service synchronization, ready to provide service.
>Dec 02 19:01:22 [49066] virt-038.cluster-qe.lab.eng.brq.redhat.com corosync notice  [CFG   ] Config reload requested by node 1
>Dec 02 19:01:22 [49066] virt-038.cluster-qe.lab.eng.brq.redhat.com corosync info    [KNET  ] pmtud: PMTUD link change for host: 6 link: 0 from 1365 to 1394
>Dec 02 19:01:22 [49066] virt-038.cluster-qe.lab.eng.brq.redhat.com corosync info    [KNET  ] pmtud: PMTUD link change for host: 5 link: 0 from 1365 to 1394
>Dec 02 19:01:22 [49066] virt-038.cluster-qe.lab.eng.brq.redhat.com corosync info    [KNET  ] pmtud: PMTUD link change for host: 4 link: 0 from 1365 to 1394
>Dec 02 19:01:22 [49066] virt-038.cluster-qe.lab.eng.brq.redhat.com corosync info    [KNET  ] pmtud: PMTUD link change for host: 3 link: 0 from 1365 to 1394
>Dec 02 19:01:22 [49066] virt-038.cluster-qe.lab.eng.brq.redhat.com corosync info    [KNET  ] pmtud: PMTUD link change for host: 2 link: 0 from 1365 to 1394
>Dec 02 19:01:22 [49066] virt-038.cluster-qe.lab.eng.brq.redhat.com corosync info    [KNET  ] pmtud: Global data MTU changed to: 1394
>[root@virt-038 ~]# sed -Ei 's/(crypto_cipher: ).*/\1aes256/' /etc/corosync/corosync.conf
>[root@virt-038 ~]# for node in $NODES; do scp /etc/corosync/corosync.conf $node:/etc/corosync/; done
>corosync.conf                                                                                                                                                                    100%  826   735.8KB/s   00:00    
>corosync.conf                                                                                                                                                                    100%  826   650.2KB/s   00:00    
>corosync.conf                                                                                                                                                                    100%  826   423.6KB/s   00:00    
>corosync.conf                                                                                                                                                                    100%  826   717.5KB/s   00:00    
>corosync.conf                                                                                                                                                                    100%  826   398.6KB/s   00:00    
>[root@virt-038 ~]# corosync-cfgtool -R
>Reloading corosync.conf...
>Done
>[root@virt-038 ~]# tail -f /var/log/cluster/corosync.log
>Dec 02 19:01:22 [49066] virt-038.cluster-qe.lab.eng.brq.redhat.com corosync info    [KNET  ] pmtud: PMTUD link change for host: 2 link: 0 from 1365 to 1394
>Dec 02 19:01:22 [49066] virt-038.cluster-qe.lab.eng.brq.redhat.com corosync info    [KNET  ] pmtud: Global data MTU changed to: 1394
>Dec 02 19:02:58 [49066] virt-038.cluster-qe.lab.eng.brq.redhat.com corosync notice  [CFG   ] Config reload requested by node 1
>Dec 02 19:02:58 [49066] virt-038.cluster-qe.lab.eng.brq.redhat.com corosync info    [TOTEM ] Configuring link 0
>Dec 02 19:02:58 [49066] virt-038.cluster-qe.lab.eng.brq.redhat.com corosync info    [TOTEM ] Configured link number 0: local addr: 2620:52:0:25a4:1800:ff:fe00:26, port=5405
>Dec 02 19:02:58 [49066] virt-038.cluster-qe.lab.eng.brq.redhat.com corosync info    [TOTEM ] kronosnet crypto reconfigured on index 1: nss/aes256/sha256
>Dec 02 19:03:00 [49066] virt-038.cluster-qe.lab.eng.brq.redhat.com corosync info    [KNET  ] link: host: 2 link: 0 is down
>Dec 02 19:03:00 [49066] virt-038.cluster-qe.lab.eng.brq.redhat.com corosync info    [KNET  ] host: host: 2 (passive) best link: 0 (pri: 1)
>Dec 02 19:03:00 [49066] virt-038.cluster-qe.lab.eng.brq.redhat.com corosync warning [KNET  ] host: host: 2 has no active links
>Dec 02 19:03:02 [49066] virt-038.cluster-qe.lab.eng.brq.redhat.com corosync notice  [TOTEM ] Token has not been received in 4219 ms


Result: After change of token.crypto_cipher to "none" and issue the reload command, problem of spamming doesn't occur. Changing the token.crypto_cipher back to "aes256" keeps the cluster in a sane state too.

Comment 6 errata-xmlrpc 2021-05-18 15:26:09 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (corosync bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1780