Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
DescriptionRaoul Scarazzini
2015-12-24 10:18:49 UTC
When doing a "pcs resource cleanup nova-compute-clone" the following failed action
never gets removed. This is an older failed action and somehow pcs does not manage
to remove it:
Failed Actions:
* nova-compute_monitor_10000 on overcloud-novacompute-3 'not running' (7): call=1577, status=complete, exitreason='none',
last-rc-change='Thu Dec 24 04:53:51 2015', queued=0ms, exec=0ms
Note that the nova-compute-clone service running on overcloud-novacompute-3 is correctly
started, we're only talking about a previous failed action that does not get cleaned up:
[heat-admin@overcloud-controller-1 logs]$ sudo pcs status|grep -A1 nova-compute-clone
Clone Set: nova-compute-clone [nova-compute]
Started: [ overcloud-novacompute-0 overcloud-novacompute-1 overcloud-novacompute-2 overcloud-novacompute-3 ]
And on the specific node:
[root@overcloud-novacompute-3 ~]# /usr/lib/ocf/resource.d/openstack/NovaCompute monitor
DEBUG: default monitor : 0
pacemaker-1.1.13-10.el7.x86_64
pcs-0.9.143-15.el7.x86_64
We attach the following files:
1) CIB
2) pcsd log from controller-0 where we ran the commands
3) corosync.log from all three nodes
http://file.rdu.redhat.com/~rscarazz/20151224_failed_resource_cleanup/
Focusing on the 04:55:02 cleanup in the logs, the resource's fail-count is correctly removed for all the compute nodes, but the resource's operation history (which "pcs status" uses to determine failed actions) is cleared on all except overcloud-novacompute-3.
This is likely a bug, but exactly where eludes me. A couple of questions:
The cleanup command will print messages like "Cleaning up nova-compute on overcloud-novacompute-0". Do you remember if it displayed such a message for overcloud-novacompute-3?
Are you able to reproduce the issue?
IIRC the cleanup message was displayed for all the overcloud-novacomputes-* resources.
I think that reproducing this specific issue is very hard. It should (and not could) be possible by using the old resource agent of the NovaCompute, but I don't have an environment available at the moment to do this. I'll keep an eye to see if I got it in some of my tests that I'm still doing.
Realistically, without a reproducer we aren't going to be able to doing anything with this. There have been multiple fixes since this time relating to clean-up, so there's a good chance it's already been addressed.
When doing a "pcs resource cleanup nova-compute-clone" the following failed action never gets removed. This is an older failed action and somehow pcs does not manage to remove it: Failed Actions: * nova-compute_monitor_10000 on overcloud-novacompute-3 'not running' (7): call=1577, status=complete, exitreason='none', last-rc-change='Thu Dec 24 04:53:51 2015', queued=0ms, exec=0ms Note that the nova-compute-clone service running on overcloud-novacompute-3 is correctly started, we're only talking about a previous failed action that does not get cleaned up: [heat-admin@overcloud-controller-1 logs]$ sudo pcs status|grep -A1 nova-compute-clone Clone Set: nova-compute-clone [nova-compute] Started: [ overcloud-novacompute-0 overcloud-novacompute-1 overcloud-novacompute-2 overcloud-novacompute-3 ] And on the specific node: [root@overcloud-novacompute-3 ~]# /usr/lib/ocf/resource.d/openstack/NovaCompute monitor DEBUG: default monitor : 0 pacemaker-1.1.13-10.el7.x86_64 pcs-0.9.143-15.el7.x86_64 We attach the following files: 1) CIB 2) pcsd log from controller-0 where we ran the commands 3) corosync.log from all three nodes http://file.rdu.redhat.com/~rscarazz/20151224_failed_resource_cleanup/