Currently, only quorum configuration can be changed with 'pcs quorum' commands.

Changing links configuration has been implemented in bz1667058

Honza, we would like to allow users to change as many configuration options as possible without stopping corosync. Therefore I need a list of corocync.conf option which can be changed in runtime. I found this list [1] but it is quite outdated (not covering knet at all). I'm aware of bz#1856397 but it only covers knet crypto options. I was unable to find any up-to-date source describing all the options. I'm primarily interested in options in totem section of corosync.conf as other sections (quorum, nodelist, totem.interface) are already covered in existing pcs commands. Do you have this documented somewhere or could you provide this information? Thanks.

[1] https://github.com/corosync/corosync/wiki/Config-file-values

@Onder,
Hi. Sure. The link you've sent (https://github.com/corosync/corosync/wiki/Config-file-values) is actually almost correct. Definitive source of all available options is corosync.conf(5). According source, all options should be changeable during runtime with exception of:
totem.version
totem.threads (not used at all - to remove)
totem.ip_version
totem.rrp_mode (not used at all - to remove)
totem.netmtu
totem.interface.ringnumber (no used at all - to remove)
totem.interface.bindnetaddr (not used when nodelist is)
totem.interface.mcastaddr (not used when nodelist is)
totem.interface.broadcast (not used when nodelist is)
totem.interface.mcastport
totem.interface.ttl
totem.transport
totem.cluster_name
quorum.provider
system.move_to_root_cgroup
system.sched_rr
system.priority
system.qb_ipc_type
system.state_dir

@Chrissie:
Aren't you aware about any other option which is not changeable in current master?

Yes, you can't change 

totem.interface.<x>.knet_transport

on the fly. You need to delete the link and re-create it.

*** Bug 1173346 has been marked as a duplicate of this bug. ***

Created attachment 1739694 [details]
proposed fix + tests

New command added:
* pcs cluster config update [transport <transport options>] [compression
  <compression options>] [crypto <crypto options>] [totem <totem options>]
  [--corosync_conf <path>]

Test:
pcs cluster config update transport knet_pmtud_interval=35 compression level=9 model=zlib crypto hash=sha512 totem token=10000 join=100
pcs cluster config update crypto cipher= hash= totem token=3000

Test:

[root@r8-node-01 ~]# rpm -q pcs
pcs-0.10.7-3.el8.x86_64

[root@r8-node-01 ~]# pcs cluster config
Cluster Name: HACluster
Transport: knet
Nodes:
  r8-node-01 (nodeid: 1)
    r8-node-01 (link: 0)
  r8-node-02 (nodeid: 2)
    r8-node-02 (link: 0)
Crypto Options:
  cipher: aes256
  hash: sha256
[root@r8-node-01 ~]# pcs cluster config update transport knet_pmtud_interval=60 compression level=9 model=zlib threshold=100 crypto cipher=aes128 hash=sha512 model=nss totem token=10000 downcheck=2000 join=60
Sending updated corosync.conf to nodes...
r8-node-01: Succeeded
r8-node-02: Succeeded
r8-node-01: Corosync configuration reloaded
[root@r8-node-01 ~]# pcs cluster config
Cluster Name: HACluster
Transport: knet
Nodes:
  r8-node-01 (nodeid: 1)
    r8-node-01 (link: 0)
  r8-node-02 (nodeid: 2)
    r8-node-02 (link: 0)
Transport Options:
  knet_pmtud_interval: 60
Compression Options:
  level: 9
  model: zlib
  threshold: 100
Crypto Options:
  cipher: aes128
  hash: sha512
  model: nss
Totem Options:
  downcheck: 2000
  join: 60
  token: 10000

[root@r8-node-01 ~]# pcs cluster config update crypto hash=
Error: If crypto option 'cipher' is enabled, crypto option 'hash' must be enabled as well
Error: Errors have occurred, therefore pcs is unable to continue
[root@r8-node-01 ~]# pcs cluster config update crypto hash= cipher=
Sending updated corosync.conf to nodes...
r8-node-02: Succeeded
r8-node-01: Succeeded
r8-node-01: Corosync configuration reloaded
[root@r8-node-01 ~]# pcs cluster config 
Cluster Name: HACluster
Transport: knet
Nodes:
  r8-node-01 (nodeid: 1)
    r8-node-01 (link: 0)
  r8-node-02 (nodeid: 2)
    r8-node-02 (link: 0)
Transport Options:
  knet_pmtud_interval: 60
Compression Options:
  level: 9
  model: zlib
  threshold: 100
Crypto Options:
  model: nss
Totem Options:
  downcheck: 2000
  join: 60
  token: 10000

Some options can't be changed during runtime:

[root@r8-node-01 ~]# pcs cluster config update transport ip_version=ipv4
Checking corosync is not running on nodes...
Error: r8-node-02: corosync is running
Error: r8-node-01: corosync is running
Error: Errors have occurred, therefore pcs is unable to continue

AFTER:
======

[root@virt-025 ~]# rpm -q pcs corosync
pcs-0.10.7-3.el8.x86_64
corosync-3.1.0-3.el8.x86_64


## Changing ip_version - this option can't be changed in the running cluster

[root@virt-025 ~]# pcs cluster config update transport ip_version=ipv4
Checking corosync is not running on nodes...
Error: virt-025: corosync is running
Error: virt-026: corosync is running
Error: Errors have occurred, therefore pcs is unable to continue

> OK: Error message is telling that the corosync is running


[root@virt-025 ~]# pcs cluster stop --all
virt-026: Stopping Cluster (pacemaker)...
virt-025: Stopping Cluster (pacemaker)...
virt-025: Stopping Cluster (corosync)...
virt-026: Stopping Cluster (corosync)...

[root@virt-025 ~]# pcs cluster config
Cluster Name: STSRHTS7522
Transport: knet
Nodes:
  virt-025 (nodeid: 1)
    virt-025 (link: 0)
  virt-026 (nodeid: 2)
    virt-026 (link: 0)
Crypto Options:
  cipher: aes256
  hash: sha256

[root@virt-025 ~]# pcs cluster config update transport ip_version=ipv4
Checking corosync is not running on nodes...
virt-025: corosync is not running
virt-026: corosync is not running
Sending updated corosync.conf to nodes...
virt-025: Succeeded
virt-026: Succeeded
[root@virt-025 ~]# echo $?
0

> OK

[root@virt-025 ~]# pcs cluster config
Cluster Name: STSRHTS7522
Transport: knet
Nodes:
  virt-025 (nodeid: 1)
    virt-025 (link: 0)
  virt-026 (nodeid: 2)
    virt-026 (link: 0)
Transport Options:
  ip_version: ipv4
Crypto Options:
  cipher: aes256
  hash: sha256

> OK: The option is in the cluster config

[root@virt-025 ~]# cat /etc/corosync/corosync.conf | grep ip_version
    ip_version: ipv4

> OK: The option is in the corosync.conf

[root@virt-025 ~]# pcs cluster start --all
virt-025: Starting Cluster...
virt-026: Starting Cluster...

[root@virt-025 ~]# corosync-cmapctl | grep ip_version
totem.ip_version (str) = ipv4

> OK


## Functionality check

[root@virt-025 ~]# cat /var/log/messages | grep 10.37.166.152,
Jan 11 09:59:45 virt-025 corosync[592245]:  [TOTEM ] Configured link number 0: local addr: 10.37.166.152, port=5405

> OK: Corosync is using ipv4 address

[root@virt-025 ~]# pcs cluster stop --all
virt-025: Stopping Cluster (pacemaker)...
virt-026: Stopping Cluster (pacemaker)...
virt-026: Stopping Cluster (corosync)...
virt-025: Stopping Cluster (corosync)...

[root@virt-025 ~]# pcs cluster config update transport ip_version=ipv6
Checking corosync is not running on nodes...
virt-025: corosync is not running
virt-026: corosync is not running
Sending updated corosync.conf to nodes...
virt-025: Succeeded
virt-026: Succeeded

[root@virt-025 ~]# pcs cluster config | grep ip_version
  ip_version: ipv6
[root@virt-025 ~]# cat /etc/corosync/corosync.conf | grep ip_version
    ip_version: ipv6
[root@virt-025 ~]# pcs cluster start --all
virt-025: Starting Cluster...
virt-026: Starting Cluster...
[root@virt-025 ~]# corosync-cmapctl | grep ip_version
totem.ip_version (str) = ipv6

> OK

[root@virt-025 ~]# cat /var/log/messages | grep 2620:52:0:25a4:1800:ff:fe00:19,
Jan 11 10:27:25 virt-025 corosync[593763]:  [TOTEM ] Configured link number 0: local addr: 2620:52:0:25a4:1800:ff:fe00:19, port=5405

> OK: Corosync is using ipv6 address


## Changing other transport options

[root@virt-025 ~]# pcs cluster config
Cluster Name: STSRHTS7522
Transport: knet
Nodes:
  virt-025 (nodeid: 1)
    virt-025 (link: 0)
  virt-026 (nodeid: 2)
    virt-026 (link: 0)
Transport Options:
  ip_version: ipv6
Crypto Options:
  cipher: aes256
  hash: sha256

[root@virt-025 ~]# pcs cluster config update transport knet_pmtud_interval=20 link_mode=rr
Sending updated corosync.conf to nodes...
virt-025: Succeeded
virt-026: Succeeded
virt-025: Corosync configuration reloaded
[root@virt-025 ~]# echo $?
0

[root@virt-025 ~]# pcs cluster config | grep "Transport Options" -A 3
Transport Options:
  ip_version: ipv6
  knet_pmtud_interval: 20
  link_mode: rr
[root@virt-025 ~]# cat /etc/corosync/corosync.conf | grep "knet_pmtud_interval\|link_mode"
    knet_pmtud_interval: 20
    link_mode: rr
[root@virt-025 ~]# corosync-cmapctl | grep "knet_pmtud_interval\|link_mode"
runtime.config.totem.knet_pmtud_interval (u32) = 20
totem.knet_pmtud_interval (u32) = 20
totem.link_mode (str) = rr

> OK: The options were changed


[root@virt-025 ~]# pcs cluster config update transport knet_pmtud_interval=25 link_mode=passive
Sending updated corosync.conf to nodes...
virt-025: Succeeded
virt-026: Succeeded
virt-025: Corosync configuration reloaded

[root@virt-025 ~]# pcs cluster config | grep "Transport Options" -A 3
Transport Options:
  ip_version: ipv6
  knet_pmtud_interval: 25
  link_mode: passive
[root@virt-025 ~]# cat /etc/corosync/corosync.conf | grep "knet_pmtud_interval\|link_mode"
    knet_pmtud_interval: 25
    link_mode: passive
[root@virt-025 ~]# corosync-cmapctl | grep "knet_pmtud_interval\|link_mode"
runtime.config.totem.knet_pmtud_interval (u32) = 25
totem.knet_pmtud_interval (u32) = 25
totem.link_mode (str) = passive

> OK: The options with previous existing setting were changed


## Changing Compression options

[root@virt-025 ~]# pcs cluster config
Cluster Name: STSRHTS7522
Transport: knet
Nodes:
  virt-025 (nodeid: 1)
    virt-025 (link: 0)
  virt-026 (nodeid: 2)
    virt-026 (link: 0)
Transport Options:
  ip_version: ipv6
  knet_pmtud_interval: 25
  link_mode: passive
Crypto Options:
  cipher: aes256
  hash: sha256

[root@virt-025 ~]# pcs cluster config update compression level=5 model=bzip2 threshold=1
Sending updated corosync.conf to nodes...
virt-025: Succeeded
virt-026: Succeeded
virt-025: Corosync configuration reloaded
[root@virt-025 ~]# echo $?
0

> OK

[root@virt-025 ~]# pcs cluster config | grep Compression -A 3
Compression Options:
  level: 5
  model: bzip2
  threshold: 1
[root@virt-025 ~]# cat /etc/corosync/corosync.conf | grep compression
    knet_compression_level: 5
    knet_compression_model: bzip2
    knet_compression_threshold: 1
[root@virt-025 ~]# corosync-cmapctl | grep "compression"
runtime.config.totem.knet_compression_level (i32) = 5
runtime.config.totem.knet_compression_model (str) = bzip2
runtime.config.totem.knet_compression_threshold (u32) = 1
totem.knet_compression_level (i32) = 5
totem.knet_compression_model (str) = bzip2
totem.knet_compression_threshold (u32) = 1

> OK


[root@virt-025 ~]# pcs cluster config update compression level=6 model=zlib threshold=150
Sending updated corosync.conf to nodes...
virt-025: Succeeded
virt-026: Succeeded
virt-025: Corosync configuration reloaded

[root@virt-025 ~]# pcs cluster config | grep Compression -A 3
Compression Options:
  level: 6
  model: zlib
  threshold: 150
[root@virt-025 ~]# cat /etc/corosync/corosync.conf | grep compression
    knet_compression_level: 6
    knet_compression_model: zlib
    knet_compression_threshold: 150
[root@virt-025 ~]# corosync-cmapctl | grep "compression"
runtime.config.totem.knet_compression_level (i32) = 6
runtime.config.totem.knet_compression_model (str) = zlib
runtime.config.totem.knet_compression_threshold (u32) = 150
totem.knet_compression_level (i32) = 6
totem.knet_compression_model (str) = zlib
totem.knet_compression_threshold (u32) = 150

> OK: The options with previous existing setting were changed


## Changing Crypto options

[root@virt-025 ~]# pcs cluster config
Cluster Name: STSRHTS7522
Transport: knet
Nodes:
  virt-025 (nodeid: 1)
    virt-025 (link: 0)
  virt-026 (nodeid: 2)
    virt-026 (link: 0)
Transport Options:
  ip_version: ipv6
  knet_pmtud_interval: 25
  link_mode: passive
Compression Options:
  level: 6
  model: zlib
  threshold: 150
Crypto Options:
  cipher: aes256
  hash: sha256

[root@virt-025 ~]# pcs cluster config update crypto cipher=aes192 hash=md5 model=openssl
Sending updated corosync.conf to nodes...
virt-025: Succeeded
virt-026: Succeeded
virt-025: Corosync configuration reloaded
[root@virt-025 ~]# echo $?
0

> OK

[root@virt-025 ~]# pcs cluster config | grep Crypto -A 3
Crypto Options:
  cipher: aes192
  hash: md5
  model: openssl
[root@virt-025 ~]# cat /etc/corosync/corosync.conf | grep crypto
    crypto_cipher: aes192
    crypto_hash: md5
    crypto_model: openssl
[root@virt-025 ~]# corosync-cmapctl | grep crypto
totem.crypto_cipher (str) = aes192
totem.crypto_hash (str) = md5
totem.crypto_model (str) = openssl

> OK


[root@virt-025 ~]# pcs cluster config update crypto cipher=aes128 hash=sha1 model=nss
Sending updated corosync.conf to nodes...
virt-025: Succeeded
virt-026: Succeeded
virt-025: Corosync configuration reloaded

[root@virt-025 ~]# pcs cluster config | grep Crypto -A 3
Crypto Options:
  cipher: aes128
  hash: sha1
  model: nss
[root@virt-025 ~]# cat /etc/corosync/corosync.conf | grep crypto
    crypto_cipher: aes128
    crypto_hash: sha1
    crypto_model: nss
[root@virt-025 ~]# corosync-cmapctl | grep crypto
totem.crypto_cipher (str) = aes128
totem.crypto_hash (str) = sha1
totem.crypto_model (str) = nss

> OK: The options with previous existing setting were changed


## Removing options from the config

[root@virt-025 ~]# pcs cluster config
Cluster Name: STSRHTS7522
Transport: knet
Nodes:
  virt-025 (nodeid: 1)
    virt-025 (link: 0)
  virt-026 (nodeid: 2)
    virt-026 (link: 0)
Transport Options:
  ip_version: ipv6
  knet_pmtud_interval: 25
  link_mode: passive
Compression Options:
  level: 6
  model: zlib
  threshold: 150
Crypto Options:
  cipher: aes128
  hash: sha1
  model: nss

[root@virt-025 ~]# pcs cluster config update crypto cipher= hash= model= transport knet_pmtud_interval= link_mode= compression level= model= threshold=
Sending updated corosync.conf to nodes...
virt-025: Succeeded
virt-026: Succeeded
virt-025: Corosync configuration reloaded

[root@virt-025 ~]# pcs cluster config
Cluster Name: STSRHTS7522
Transport: knet
Nodes:
  virt-025 (nodeid: 1)
    virt-025 (link: 0)
  virt-026 (nodeid: 2)
    virt-026 (link: 0)
Transport Options:
  ip_version: ipv6

> OK


## Changing options from multiple categories at once

[root@virt-025 ~]# pcs cluster config update transport knet_pmtud_interval=25 compression level=5 model=zlib
Sending updated corosync.conf to nodes...
virt-025: Succeeded
virt-026: Succeeded
virt-025: Corosync configuration reloaded
[root@virt-025 ~]# echo $?
0

[root@virt-025 ~]# pcs cluster config
Cluster Name: STSRHTS7522
Transport: knet
Nodes:
  virt-025 (nodeid: 1)
    virt-025 (link: 0)
  virt-026 (nodeid: 2)
    virt-026 (link: 0)
Transport Options:
  ip_version: ipv6
  knet_pmtud_interval: 25
Compression Options:
  level: 5
  model: zlib

> OK

[root@virt-025 ~]# pcs cluster config update transport knet_pmtud_interval= compression level= model=
Sending updated corosync.conf to nodes...
virt-025: Succeeded
virt-026: Succeeded
virt-025: Corosync configuration reloaded


## Changing options in specified corosync_conf file

[root@virt-025 ~]# cp /etc/corosync/corosync.conf my_conf.conf
[root@virt-025 ~]# pcs cluster config update transport knet_pmtud_interval=25 compression level=5 model=zlib --corosync_conf my_conf.conf 
[root@virt-025 ~]# echo $?
0
[root@virt-025 ~]# cat my_conf.conf | grep 'knet_pmtud_interval\|level\|model'
    knet_pmtud_interval: 25
    knet_compression_level: 5
    knet_compression_model: zlib

> OK: The options are changed in the specified file

[root@virt-025 ~]# cat /etc/corosync/corosync.conf | grep 'knet_pmtud_interval\|level\|model'
[root@virt-025 ~]# echo $?
1
[root@virt-025 ~]# pcs cluster config | grep 'knet_pmtud_interval\|level\|model'
[root@virt-025 ~]# echo $?
1
[root@virt-025 ~]# corosync-cmapctl | grep 'knet_pmtud_interval\|level\|model'
runtime.config.totem.knet_compression_level (i32) = 0
runtime.config.totem.knet_compression_model (str) = none
runtime.config.totem.knet_pmtud_interval (u32) = 30

> OK: Changed options are only in the specified file


## Invalid inputs

# Setting option with wrong value

[root@virt-025 ~]# pcs cluster config update crypto hash=test
Error: 'test' is not a valid hash value, use 'md5', 'none', 'sha1', 'sha256', 'sha384', 'sha512'
Error: Errors have occurred, therefore pcs is unable to continue
[root@virt-025 ~]# echo $? 
1
[root@virt-025 ~]# pcs cluster config | grep hash
[root@virt-025 ~]# echo $?
1

> OK

[root@virt-025 ~]# pcs cluster config update transport knet_pmtud_interval=-5
Error: '-5' is not a valid knet_pmtud_interval value, use a non-negative integer
Error: Errors have occurred, therefore pcs is unable to continue
[root@virt-025 ~]# echo $?
1
[root@virt-025 ~]# pcs cluster config | grep knet_pmtud_interval
[root@virt-025 ~]# echo $?
1

> OK


# Setting option without fulfilled dependence on other option

root@virt-025 ~]# pcs cluster config update crypto cipher=aes256
Error: If crypto option 'cipher' is enabled, crypto option 'hash' must be enabled as well
Error: Errors have occurred, therefore pcs is unable to continue
[root@virt-025 ~]# echo $?
1
[root@virt-025 ~]# pcs cluster config | grep cipher
[root@virt-025 ~]# echo $?
1

> OK


# Setting option under wrong category

[root@virt-025 ~]# pcs cluster config update compression knet_pmtud_interval=10
Error: invalid compression option 'knet_pmtud_interval', allowed options are: 'level', 'model', 'threshold'
Error: Errors have occurred, therefore pcs is unable to continue
[root@virt-025 ~]# echo $?
1
[root@virt-025 ~]# pcs cluster config | grep knet_pmtud_interval
[root@virt-025 ~]# echo $?
1

> OK

[root@virt-025 ~]# pcs cluster config update transport knet_pmtud_interval=10 totem hash=sha1
Error: invalid totem option 'hash', allowed options are: 'consensus', 'downcheck', 'fail_recv_const', 'heartbeat_failures_allowed', 'hold', 'join', 'max_messages', 'max_network_delay', 'merge', 'miss_count_const', 'send_join', 'seqno_unchanged_const', 'token', 'token_coefficient', 'token_retransmit', 'token_retransmits_before_loss_const', 'window_size'
Error: Errors have occurred, therefore pcs is unable to continue
[root@virt-025 ~]# echo $?
1
[root@virt-025 ~]# pcs cluster config | grep knet_pmtud_interval
[root@virt-025 ~]# echo $?
1
[root@virt-025 ~]# pcs cluster config | grep hash
[root@virt-025 ~]# echo $?
1

> OK


# Setting option without category

[root@virt-025 ~]# pcs cluster config update knet_pmtud_interval=10
{...}
\usage output
{...}
[root@virt-025 ~]# echo $?
1
[root@virt-025 ~]# pcs cluster config | grep knet_pmtud_interval
[root@virt-025 ~]# echo $?
1

> OK


# Inserting invalid option

[root@virt-025 ~]# pcs cluster config update --test
{...}
\usage output
{...}
option --test not recognized
[root@virt-025 ~]# echo $?
1

> OK


# Setting non-existent corosync_conf file

[root@virt-025 ~]# pcs cluster config update --corosync_conf temp.conf 
Error: Unable to read Corosync configuration 'temp.conf': No such file or directory: 'temp.conf'
[root@virt-025 ~]# echo $?
1

> OK


# Setting compression and crypto options with udp transport (tested on different cluster with udp transport set)

[root@virt-022 ~]# rpm -q pcs
pcs-0.10.7-3.el8.x86_64
[root@virt-022 ~]# pcs cluster config | grep Transport
Transport: udp

[root@virt-022 ~]# pcs cluster config update compression level=5
Error: The udp/udpu transport does not support 'compression' options, use 'knet' transport
Error: Errors have occurred, therefore pcs is unable to continue
[root@virt-022 ~]# echo $?
1
[root@virt-022 ~]# pcs cluster config | grep level
[root@virt-022 ~]# echo $?
1

> OK

[root@virt-022 ~]# pcs cluster config update crypto hash=sha256
Error: The udp/udpu transport does not support 'crypto' options, use 'knet' transport
Error: Errors have occurred, therefore pcs is unable to continue
[root@virt-022 ~]# echo $?
1
[root@virt-022 ~]# pcs cluster config | grep hash
[root@virt-022 ~]# echo $?
1

> OK


# Omitting options (not an invalid case)

[root@virt-025 ~]# pcs cluster config update
Sending updated corosync.conf to nodes...
virt-025: Succeeded
virt-026: Succeeded
virt-025: Corosync configuration reloaded
[root@virt-025 ~]# echo $?
0
[root@virt-025 ~]# pcs cluster config
Cluster Name: STSRHTS7522
Transport: knet
Nodes:
  virt-025 (nodeid: 1)
    virt-025 (link: 0)
  virt-026 (nodeid: 2)
    virt-026 (link: 0)
Transport Options:
  ip_version: ipv6

[root@virt-025 ~]# pcs cluster config update totem transport
Sending updated corosync.conf to nodes...
virt-025: Succeeded
virt-026: Succeeded
virt-025: Corosync configuration reloaded
[root@virt-025 ~]# echo $?
0
[root@virt-025 ~]# pcs cluster config
Cluster Name: STSRHTS7522
Transport: knet
Nodes:
  virt-025 (nodeid: 1)
    virt-025 (link: 0)
  virt-026 (nodeid: 2)
    virt-026 (link: 0)
Transport Options:
  ip_version: ipv6

> Omitting a category or an option under a category is allowed as no setting will change


Additional tests will follow.

## Changing totem options

[root@virt-486 ~]# rpm -q pcs && rpm -q corosync
pcs-0.10.7-3.el8.x86_64
corosync-3.1.0-3.el8.x86_64

# Default configuration, totem token is set to 3s

[root@virt-486 ~]# pcs cluster config
Cluster Name: STSRHTS14540
Transport: knet
Nodes:
  virt-486 (nodeid: 1)
    virt-486 (link: 0)
  virt-487 (nodeid: 2)
    virt-487 (link: 0)
Crypto Options:
  cipher: aes256
  hash: sha256
  
[root@virt-486 ~]# pcs cluster corosync | grep -A6 totem
totem {
    version: 2
    cluster_name: STSRHTS14540
    transport: knet
    crypto_cipher: aes256
    crypto_hash: sha256
}

[root@virt-486 ~]# corosync-cmapctl | grep totem.token
runtime.config.totem.token (u32) = 3000
runtime.config.totem.token_retransmit (u32) = 714
runtime.config.totem.token_retransmits_before_loss_const (u32) = 4
runtime.config.totem.token_warning (u32) = 75


# Block corosync port on one of the nodes 
# Check the logs

[root@virt-486 ~]# tail -f /var/log/messages
...
Jan 11 12:25:28 virt-487 corosync[49822]:  [TOTEM ] Token has not been received in 2250 ms
Jan 11 12:25:29 virt-487 corosync[49822]:  [TOTEM ] A processor failed, forming new configuration: token timed out (3000ms), waiting 3600ms for consensus.
...

> Results in fencing after token times out


# Unblock corosync
# Raise totem token to 5s 

[root@virt-486 ~]# pcs cluster config update totem token=5000
Sending updated corosync.conf to nodes...
virt-486: Succeeded
virt-487: Succeeded
virt-486: Corosync configuration reloaded
[root@virt-486 ~]# echo $?
0

[root@virt-486 ~]# pcs cluster config | grep token
  token: 5000

[root@virt-486 ~]# pcs cluster corosync | grep -A7 totem
totem {
    version: 2
    cluster_name: STSRHTS14540
    transport: knet
    crypto_cipher: aes256
    crypto_hash: sha256
    token: 5000
}

[root@virt-486 ~]#  corosync-cmapctl | grep totem.token
runtime.config.totem.token (u32) = 5000
runtime.config.totem.token_retransmit (u32) = 1190
runtime.config.totem.token_retransmits_before_loss_const (u32) = 4
runtime.config.totem.token_warning (u32) = 75
totem.token (u32) = 5000

> OK, configuration and corosync.conf have changed


# Block corosync port on one of the nodes 
# Check the logs

[root@virt-486 ~]# tail -f /var/log/messages
...
Jan 11 16:57:51 virt-486 corosync[49682]:  [TOTEM ] Token has not been received in 3750 ms
Jan 11 16:57:53 virt-486 corosync[49682]:  [TOTEM ] A processor failed, forming new configuration: token timed out (5000ms), waiting 6000ms for consensus.
...

> The change of token is apparent from the runtime configuration, logs and prolonged time for fencing to be triggered


# Unblock corosync 
# Set token to the original value and try to update other parameters

[root@virt-486 ~]# pcs cluster config update totem consensus=3000 downcheck=1200 fail_recv_const=3000 heartbeat_failures_allowed=1 hold=200 join=80 max_messages=15 max_network_delay=70 merge=150 miss_count_const=0 send_join=1 seqno_unchanged_const=20 token=3000 token_coefficient=800 token_retransmit=250 token_retransmits_before_loss_const=5 window_size=25
Sending updated corosync.conf to nodes...
virt-486: Succeeded
virt-487: Succeeded
virt-486: Corosync configuration reloaded
[root@virt-486 ~]# echo $?
0

[root@virt-486 ~]# pcs cluster config
Cluster Name: STSRHTS14540
Transport: knet
Nodes:
  virt-486 (nodeid: 1)
    virt-486 (link: 0)
  virt-487 (nodeid: 2)
    virt-487 (link: 0)
Crypto Options:
  cipher: aes256
  hash: sha256
Totem Options:
  consensus: 3000
  downcheck: 1200
  fail_recv_const: 3000
  heartbeat_failures_allowed: 1
  hold: 200
  join: 80
  max_messages: 15
  max_network_delay: 70
  merge: 150
  miss_count_const: 0
  send_join: 1
  seqno_unchanged_const: 20
  token: 3000
  token_coefficient: 800
  token_retransmit: 250
  token_retransmits_before_loss_const: 5
  window_size: 25

[root@virt-486 ~]# pcs cluster corosync | grep -A23 totem
totem {
    version: 2
    cluster_name: STSRHTS14540
    transport: knet
    crypto_cipher: aes256
    crypto_hash: sha256
    token: 3000
    consensus: 3000
    downcheck: 1200
    fail_recv_const: 3000
    heartbeat_failures_allowed: 1
    hold: 200
    join: 80
    max_messages: 15
    max_network_delay: 70
    merge: 150
    miss_count_const: 0
    send_join: 1
    seqno_unchanged_const: 20
    token_coefficient: 800
    token_retransmit: 250
    token_retransmits_before_loss_const: 5
    window_size: 25
}


# Check corosync.conf from another node

[root@virt-486 ~]# pcs cluster corosync virt-487 | grep -A23 totem
totem {
    version: 2
    cluster_name: STSRHTS14540
    transport: knet
    crypto_cipher: aes256
    crypto_hash: sha256
    token: 3000
    consensus: 3000
    downcheck: 1200
    fail_recv_const: 3000
    heartbeat_failures_allowed: 1
    hold: 200
    join: 80
    max_messages: 15
    max_network_delay: 70
    merge: 150
    miss_count_const: 0
    send_join: 1
    seqno_unchanged_const: 20
    token_coefficient: 800
    token_retransmit: 250
    token_retransmits_before_loss_const: 5
    window_size: 25
}

[root@virt-486 ~]# corosync-cmapctl | grep totem
config.totemconfig_reload_in_progress (u8) = 0
runtime.config.totem.block_unlisted_ips (u32) = 1
runtime.config.totem.consensus (u32) = 3000
runtime.config.totem.downcheck (u32) = 1200
runtime.config.totem.fail_recv_const (u32) = 3000
runtime.config.totem.heartbeat_failures_allowed (u32) = 1
runtime.config.totem.hold (u32) = 200
runtime.config.totem.interface.0.knet_ping_interval (u32) = 1250
runtime.config.totem.interface.0.knet_ping_timeout (u32) = 2500
runtime.config.totem.join (u32) = 80
runtime.config.totem.knet_compression_level (i32) = 0
runtime.config.totem.knet_compression_model (str) = none
runtime.config.totem.knet_compression_threshold (u32) = 0
runtime.config.totem.knet_pmtud_interval (u32) = 30
runtime.config.totem.max_messages (u32) = 15
runtime.config.totem.max_network_delay (u32) = 70
runtime.config.totem.merge (u32) = 150
runtime.config.totem.miss_count_const (u32) = 5
runtime.config.totem.send_join (u32) = 1
runtime.config.totem.seqno_unchanged_const (u32) = 20
runtime.config.totem.token (u32) = 3000
runtime.config.totem.token_retransmit (u32) = 250
runtime.config.totem.token_retransmits_before_loss_const (u32) = 5
runtime.config.totem.token_warning (u32) = 75
runtime.config.totem.window_size (u32) = 25
totem.cluster_name (str) = STSRHTS14540
totem.consensus (u32) = 3000
totem.crypto_cipher (str) = aes256
totem.crypto_hash (str) = sha256
totem.downcheck (u32) = 1200
totem.fail_recv_const (u32) = 3000
totem.heartbeat_failures_allowed (u32) = 1
totem.hold (u32) = 200
totem.join (u32) = 80
totem.max_messages (u32) = 15
totem.max_network_delay (u32) = 70
totem.merge (u32) = 150
totem.miss_count_const (u32) = 0
totem.send_join (u32) = 1
totem.seqno_unchanged_const (u32) = 20
totem.token (u32) = 3000
totem.token_coefficient (u32) = 800
totem.token_retransmit (u32) = 250
totem.token_retransmits_before_loss_const (u32) = 5
totem.transport (str) = knet
totem.version (u32) = 2
totem.window_size (u32) = 25

> OK


# Revert the configuration to default values

[root@virt-486 ~]# pcs cluster config update totem consensus= downcheck= fail_recv_const= heartbeat_failures_allowed= hold= join= max_messages= max_network_delay= merge= miss_count_const= send_join= seqno_unchanged_const= token= token_coefficient= token_retransmit= token_retransmits_before_loss_const= window_size=
Sending updated corosync.conf to nodes...
virt-486: Succeeded
virt-487: Succeeded
virt-486: Corosync configuration reloaded
[root@virt-486 ~]# echo $?
0

[root@virt-486 ~]# pcs cluster config
Cluster Name: STSRHTS14540
Transport: knet
Nodes:
  virt-486 (nodeid: 1)
    virt-486 (link: 0)
  virt-487 (nodeid: 2)
    virt-487 (link: 0)
Crypto Options:
  cipher: aes256
  hash: sha256

[root@virt-486 ~]# pcs cluster corosync | grep -A6 totem
totem {
    version: 2
    cluster_name: STSRHTS14540
    transport: knet
    crypto_cipher: aes256
    crypto_hash: sha256
}

[root@virt-486 ~]#  corosync-cmapctl | grep totem
config.totemconfig_reload_in_progress (u8) = 0
runtime.config.totem.block_unlisted_ips (u32) = 1
runtime.config.totem.consensus (u32) = 3600
runtime.config.totem.downcheck (u32) = 1000
runtime.config.totem.fail_recv_const (u32) = 2500
runtime.config.totem.heartbeat_failures_allowed (u32) = 0
runtime.config.totem.hold (u32) = 561
runtime.config.totem.interface.0.knet_ping_interval (u32) = 750
runtime.config.totem.interface.0.knet_ping_timeout (u32) = 1500
runtime.config.totem.join (u32) = 50
runtime.config.totem.knet_compression_level (i32) = 0
runtime.config.totem.knet_compression_model (str) = none
runtime.config.totem.knet_compression_threshold (u32) = 0
runtime.config.totem.knet_pmtud_interval (u32) = 30
runtime.config.totem.max_messages (u32) = 17
runtime.config.totem.max_network_delay (u32) = 50
runtime.config.totem.merge (u32) = 200
runtime.config.totem.miss_count_const (u32) = 5
runtime.config.totem.send_join (u32) = 0
runtime.config.totem.seqno_unchanged_const (u32) = 30
runtime.config.totem.token (u32) = 3000
runtime.config.totem.token_retransmit (u32) = 714
runtime.config.totem.token_retransmits_before_loss_const (u32) = 4
runtime.config.totem.token_warning (u32) = 75
runtime.config.totem.window_size (u32) = 50
totem.cluster_name (str) = STSRHTS14540
totem.crypto_cipher (str) = aes256
totem.crypto_hash (str) = sha256
totem.transport (str) = knet
totem.version (u32) = 2

> OK


# Update the cluster configuration in a given file

[root@virt-486 ~]# cp /etc/corosync/corosync.conf /tmp/corosync.conf
[root@virt-486 ~]# pcs cluster config update totem consensus=3000 downcheck=1200 fail_recv_const=3000 heartbeat_failures_allowed=1 hold=200 join=80 max_messages=15 max_network_delay=70 merge=150 miss_count_const=0 send_join=1 seqno_unchanged_const=20 token=3000 token_coefficient=800 token_retransmit=250 token_retransmits_before_loss_const=5 window_size=25 --corosync_conf /tmp/corosync.conf 
[root@virt-486 ~]# echo $?
0

[root@virt-486 ~]# grep -A23 totem /tmp/corosync.conf
totem {
    version: 2
    cluster_name: STSRHTS14540
    transport: knet
    crypto_cipher: aes256
    crypto_hash: sha256
    token: 3000
    consensus: 3000
    downcheck: 1200
    fail_recv_const: 3000
    heartbeat_failures_allowed: 1
    hold: 200
    join: 80
    max_messages: 15
    max_network_delay: 70
    merge: 150
    miss_count_const: 0
    send_join: 1
    seqno_unchanged_const: 20
    token_coefficient: 800
    token_retransmit: 250
    token_retransmits_before_loss_const: 5
    window_size: 25
}

[root@virt-486 ~]# pcs cluster corosync | grep -A6 totem
totem {
    version: 2
    cluster_name: STSRHTS14540
    transport: knet
    crypto_cipher: aes256
    crypto_hash: sha256
}

> OK, configuration is updated only in the specified file


# Some of the totem options cannot be changed in an existing cluster

[root@virt-486 ~]# pcs cluster config update totem cluster_name=test_name
Error: invalid totem option 'cluster_name', allowed options are: 'consensus', 'downcheck', 'fail_recv_const', 'heartbeat_failures_allowed', 'hold', 'join', 'max_messages', 'max_network_delay', 'merge', 'miss_count_const', 'send_join', 'seqno_unchanged_const', 'token', 'token_coefficient', 'token_retransmit', 'token_retransmits_before_loss_const', 'window_size'
Error: Errors have occurred, therefore pcs is unable to continue
[root@virt-486 ~]# echo $?
1

> OK



Marking verified in pcs-0.10.7-3.el8.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (pcs bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2021:1737