Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Feature:
Corosync UDPU automatically sends messages to the appropriate members, and avoids sending to missing members when its not required
Reason:
When using UDPU, all messages get sent to all configured members, as opposed to only the active members. This makes sense for merge detection messages, but for everything else, it creates unnecessary traffic to missing members and may trigger excessive arp requests on the network. It is requested that corosync be modified to only send messages to missing members when needed, and otherwise only send messages to active ring members.
Customer's network team observed an excessive amount of arp requests from the cluster for a node that was not active. They wish to limit this, and can currently only do so for merge detection messages by increasing the timeout.
Result:
Most of udpu messages are sent only to active members with exception of messages required for proper detection of merge or new member (1-2 pkts/sec)
Created attachment 930986[details]
totem: Inform RRP about membership changes
totem: Inform RRP about membership changes
Services are informed about membership changes, but if same information
is needed inside totemrrp or totemnet, it's impossible to gather this
information.
Patch makes this possible for now only for RRP with empty callbacks.
Signed-off-by: Jan Friesse <jfriesse>
Reviewed-by: Christine Caulfield <ccaulfie>
Created attachment 930987[details]
totemnet: Add totemnet_member_set_active
totemnet: Add totemnet_member_set_active
totemnet_member_set_active together with transport specific
member_set_active makes possible for totemnet (and more interestingly
transport) to be informed about membership changes.
Signed-off-by: Jan Friesse <jfriesse>
Reviewed-by: Christine Caulfield <ccaulfie>
Created attachment 930989[details]
totemrrp: Implement *_membership_changed
totemrrp: Implement *_membership_changed
All *_membership_changed calls totemnet_member_set_active passing 1 as
active parameter for joined nodes and 0 for left nodes.
Signed-off-by: Jan Friesse <jfriesse>
Reviewed-by: Christine Caulfield <ccaulfie>
Created attachment 930998[details]
totemudpu: Implement member_set_active
totemudpu: Implement member_set_active
Member active is used for sending "multicast" messages only to members
of ring. This reduces network load if some nodes are intentionally down.
Only regular multicast message load is reduced (messages sent by
totemudpu_mcast_noflush_send), because special messages (like hold
cancel, join message, ...) still have to be send to all members to
ensure correct behavior.
Signed-off-by: Jan Friesse <jfriesse>
Reviewed-by: Christine Caulfield <ccaulfie>
Created attachment 931001[details]
totemudpu: Send msgs to all members occasionally
totemudpu: Send msgs to all members occasionally
To follow spec it's needed to send messages to all nodes (not only
active members) from time to time to detect merge.
This is needed in situations when totemsrp merge timer isn't running
(because there is enough messages sent by processors) to detect merge.
Example scenario:
- 3 nodes, all of them running cpgverify
- One node is isolated (iptables for example)
- Node is un-isolated
Without this commit, node will not merge as long as the cpgverify is
running.
Signed-off-by: Jan Friesse <jfriesse>
Reviewed-by: Christine Caulfield <ccaulfie>
It's impossible to achieve no packets sent to disconnected nodes together with reliable and fast merge detection. Eventho 5 patches from this BZ reduces traffic significantly (for example with cpgverify, from few thousandths pkts/sec to 1-2 pkts/sec).
This is basic scenario:
- 3 nodes (A, B, C). Configured with UDPU
- Start node A. A should start and (with debugging enabled) there is "Marking UDPU member NODE_A_IP active"
- Start node B. B should again start and there is "Marking UDPU member NODE_B_IP active" followed by "Marking UDPU member NODE_A_IP active"
- exec cpg-load on node B
- dump network trafic (tcpdump?) and see how much packets are sent to node C. There should be only few per second on node B, and no packet on node C
- now exec cpg-load on node A + dump traffic. there should be again few per second packets for node C on both nodes A/B
- stop cpg-load on node A and B - there should be 0 packets originated from node B for node C and again few per second from node A
- start corosync and cpg-load on node C
- isolate node C
- exec cpg-load on node B
- un-isolate node C
- nodes should create full membership quickly
I was also testing similar scenario with 4 nodes (isolate 1, 2 nodes or create 2 partitons) + RRP. General behavior should be:
- if ring is quiet, leader is sending msg to all nodes from time to time
- if there is traffic, node(s) generating traffic sends msh to all nodes from time to time
- merge have to be quick (may be little slower then without patches applied)
If this Feature should be included in the 7.1 Release Notes, could you please change the Doc Type from Enhancement to "Release Note"?
Note that the Release Notes are intended to list the most prominent and customer-relevant new features rather than every single enhancement.
Cheers,
Jirka
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://rhn.redhat.com/errata/RHBA-2015-0365.html
Created attachment 930986 [details] totem: Inform RRP about membership changes totem: Inform RRP about membership changes Services are informed about membership changes, but if same information is needed inside totemrrp or totemnet, it's impossible to gather this information. Patch makes this possible for now only for RRP with empty callbacks. Signed-off-by: Jan Friesse <jfriesse> Reviewed-by: Christine Caulfield <ccaulfie>