Bug 1946623

Summary: corosync crash with compression enabled
Product: Red Hat Enterprise Linux 8 Reporter: Jan Friesse <jfriesse>
Component: corosyncAssignee: Jan Friesse <jfriesse>
Status: CLOSED ERRATA QA Contact: cluster-qe <cluster-qe>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 8.4CC: ccaulfie, cfeist, cluster-maint, djansa, lmiksik, phagara
Target Milestone: betaKeywords: Triaged, ZStream
Target Release: ---Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: corosync-3.1.0-4.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1951049 (view as bug list) Environment:
Last Closed: 2021-11-09 17:49:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1951049    
Attachments:
Description Flags
Proposed patch (for 3.1.0) none

Description Jan Friesse 2021-04-06 14:20:42 UTC
Description of problem:
Corosync 3.1.0 (and 3.1.1) crashes when compression is enabled. Original upstream report https://github.com/corosync/corosync/issues/630

Version-Release number of selected component (if applicable):
3.1.0

How reproducible:
Memory corruption so 100% in valgrind, high probability otherwise

Steps to Reproduce:
1. Enable compression
2. Start corosync

Actual results:
Corosync crashes on startup

Expected results:
No crash

Additional info:
Fixed upstream 3.1.2. Corosync 3.0.x is not affected.

Comment 1 Jan Friesse 2021-04-06 14:26:17 UTC
Created attachment 1769583 [details]
Proposed patch (for 3.1.0)

Comment 5 Dean Jansa 2021-04-12 14:00:47 UTC
ON_QA bug without Verified:Tested should be in the MODIFIED state.

Comment 12 Patrik Hagara 2021-06-21 10:39:50 UTC
before fix (corosync-3.1.0-3.el8)
=================================

> # pcs host auth virt-031 virt-032 virt-033 -u hacluster
> # pcs cluster setup rhel84 virt-031 virt-032 virt-033 transport knet compression level=6 model=zlib
> # pcs cluster corosync | grep compression
>     knet_compression_level: 6
>     knet_compression_model: zlib
> 
> 
> # pcs cluster start --all
> virt-033: Starting Cluster...
> virt-031: Starting Cluster...
> virt-032: Error connecting to virt-032 - (HTTP error: 400)
> Error: unable to start all nodes
> virt-032: Error connecting to virt-032 - (HTTP error: 400)
> 
> 
> [root@virt-032]# coredumpctl gdb 103646
>            PID: 103646 (corosync)
>            UID: 0 (root)
>            GID: 0 (root)
>         Signal: 11 (SEGV)
>      Timestamp: Wed 2021-04-21 12:52:42 CEST (2min 50s ago)
>   Command Line: /usr/sbin/corosync -f
>     Executable: /usr/sbin/corosync
>  Control Group: /system.slice/corosync.service
>           Unit: corosync.service
>          Slice: system.slice
>        Boot ID: ca297a1ad52449e3b8c4832ba7c11674
>     Machine ID: 5fcf2a5e7b1a4a93bac9dbfc69c3711e
>       Hostname: virt-032.cluster-qe.lab.eng.brq.redhat.com
>        Storage: /var/lib/systemd/coredump/core.corosync.0.ca297a1ad52449e3b8c4832ba7c11674.103646.1619002362000000.lz4
>        Message: Process 103646 (corosync) of user 0 dumped core.
>                 
>                 Stack trace of thread 103646:
>                 #0  0x00007fbf37653924 __pthread_mutex_lock (libpthread.so.0)
>                 #1  0x00007fbf37a7c53c get_global_wrlock (libknet.so.1)
>                 #2  0x00007fbf37a7695c knet_handle_compress (libknet.so.1)
>                 #3  0x00005568f4a58ba0 totemknet_configure_compression (corosync)
>                 #4  0x00005568f4a59dbd totemknet_initialize (corosync)
>                 #5  0x00005568f4a49c85 totemnet_initialize (corosync)
>                 #6  0x00005568f4a55421 totemsrp_initialize (corosync)
>                 #7  0x00005568f4a56a8b totempg_initialize (corosync)
>                 #8  0x00005568f4a25ae2 main (corosync)
>                 #9  0x00007fbf372a7493 __libc_start_main (libc.so.6)
>                 #10 0x00005568f4a262fe _start (corosync)

Result: corosync on random node crashes on startup when compression is enabled.


after fix (corosync-3.1.0-5.el8)
================================

> [root@virt-140 ~]# pcs host auth virt-140 virt-141 virt-143
> Username: hacluster
> Password: 
> virt-143: Authorized
> virt-141: Authorized
> virt-140: Authorized
> [root@virt-140 ~]# pcs cluster setup foo virt-140 virt-141 virt-143 transport knet compression level=6 model=zlib
> No addresses specified for host 'virt-140', using 'virt-140'
> No addresses specified for host 'virt-141', using 'virt-141'
> No addresses specified for host 'virt-143', using 'virt-143'
> Destroying cluster on hosts: 'virt-140', 'virt-141', 'virt-143'...
> virt-140: Successfully destroyed cluster
> virt-143: Successfully destroyed cluster
> virt-141: Successfully destroyed cluster
> Requesting remove 'pcsd settings' from 'virt-140', 'virt-141', 'virt-143'
> virt-140: successful removal of the file 'pcsd settings'
> virt-143: successful removal of the file 'pcsd settings'
> virt-141: successful removal of the file 'pcsd settings'
> Sending 'corosync authkey', 'pacemaker authkey' to 'virt-140', 'virt-141', 'virt-143'
> virt-140: successful distribution of the file 'corosync authkey'
> virt-140: successful distribution of the file 'pacemaker authkey'
> virt-143: successful distribution of the file 'corosync authkey'
> virt-143: successful distribution of the file 'pacemaker authkey'
> virt-141: successful distribution of the file 'corosync authkey'
> virt-141: successful distribution of the file 'pacemaker authkey'
> Sending 'corosync.conf' to 'virt-140', 'virt-141', 'virt-143'
> virt-140: successful distribution of the file 'corosync.conf'
> virt-143: successful distribution of the file 'corosync.conf'
> virt-141: successful distribution of the file 'corosync.conf'
> Cluster has been successfully set up.
> [root@virt-140 ~]# pcs cluster corosync | grep compression
>     knet_compression_level: 6
>     knet_compression_model: zlib
> [root@virt-140 ~]# pcs cluster start --all
> virt-141: Starting Cluster...
> virt-140: Starting Cluster...
> virt-143: Starting Cluster...
> [root@virt-140 ~]# pcs status
> Cluster name: foo
> 
> WARNINGS:
> No stonith devices and stonith-enabled is not false
> 
> Cluster Summary:
>   * Stack: corosync
>   * Current DC: virt-143 (version 2.1.0-2.el8-7c3f660707) - partition with quorum
>   * Last updated: Mon Jun 21 12:38:07 2021
>   * Last change:  Mon Jun 21 12:37:48 2021 by hacluster via crmd on virt-143
>   * 3 nodes configured
>   * 0 resource instances configured
> 
> Node List:
>   * Online: [ virt-140 virt-141 virt-143 ]
> 
> Full List of Resources:
>   * No resources
> 
> Daemon Status:
>   corosync: active/disabled
>   pacemaker: active/disabled
>   pcsd: active/enabled

Result: corosync startup succeeds with compression enabled.

Comment 14 errata-xmlrpc 2021-11-09 17:49:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (corosync bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4169