Bug 1891835
| Summary: | promoting a galera node can fail due to a race in attributes update | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Damien Ciabrini <dciabrin> |
| Component: | resource-agents | Assignee: | Oyvind Albrigtsen <oalbrigt> |
| Status: | CLOSED ERRATA | QA Contact: | cluster-qe <cluster-qe> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 8.2 | CC: | agk, cluster-maint, fdinitto, michele, pkomarov |
| Target Milestone: | rc | Flags: | pm-rhel:
mirror+
|
| Target Release: | 8.4 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | resource-agents-4.1.1-72.el8 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-05-18 15:11:15 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Damien Ciabrini
2020-10-27 13:47:09 UTC
The original bug is hard to reproduce as it is a race. But the fix is really just inverting two line in the agent:
@@ -587,8 +587,8 @@ detect_first_master()
fi
ocf_log info "Promoting $best_node to be our bootstrap node"
- set_master_score $best_node
set_bootstrap_node $best_node
+ set_master_score $best_node
}
So a simple overcloud deployment is enough to verify that the original behaviour is preserved, and the new line ordering is enough to ensure that the fix will work as expected.
Verified , [stack@undercloud-0 ~]$ ansible database -b -mshell -a'podman exec `podman ps -f name=galera-bundle -q` sh -c "rpm -q resource-agents";rpm -q resource-agents' [WARNING]: Found both group and host with same name: undercloud database-0 | CHANGED | rc=0 >> resource-agents-4.1.1-79.el8.x86_64 resource-agents-4.1.1-79.el8.x86_64 database-2 | CHANGED | rc=0 >> resource-agents-4.1.1-79.el8.x86_64 resource-agents-4.1.1-79.el8.x86_64 database-1 | CHANGED | rc=0 >> resource-agents-4.1.1-79.el8.x86_64 resource-agents-4.1.1-79.el8.x86_64 #testing disruptive galera scenario: git clone -c http.sslVerify=false https://code.engineering.redhat.com/gerrit/openstack-pidone-qe ansible-playbook -v openstack-pidone-qe/CI/ansible_sts/ansible_sts_composable_roles_tests.yaml --tags 06_KILL_ALL_GALERA|& tee ansible_sts_results/06_KILL_ALL_GALERA.log ansible-playbook -v openstack-pidone-qe/CI/ansible_sts/ansible_sts_composable_roles_tests.yaml --tags 07_REMOVE_ALL_GRASTATE_GALERA|& tee ansible_sts_results/07_REMOVE_ALL_GRASTATE_GALERA.log ansible-playbook -v openstack-pidone-qe/CI/ansible_sts/ansible_sts_composable_roles_tests.yaml --tags 09_GENERATE_BIG_DATA_GALERA|& tee ansible_sts_results/09_GENERATE_BIG_DATA_GALERA.log #agents have recovered after all disruptive galera tests: * galera-bundle-0 (ocf::heartbeat:galera): Master database-2 * galera-bundle-1 (ocf::heartbeat:galera): Master database-1 * galera-bundle-2 (ocf::heartbeat:galera): Master database-0 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (resource-agents bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:1736 |