Bug 1447916
| Summary: | cleanup of a bundle resource may result in the crmd process on the DC node taking 100 % of a CPU | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Tomas Jelinek <tojeline> |
| Component: | pacemaker | Assignee: | Ken Gaillot <kgaillot> |
| Status: | CLOSED WONTFIX | QA Contact: | cluster-qe <cluster-qe> |
| Severity: | high | Docs Contact: | |
| Priority: | low | ||
| Version: | 8.3 | CC: | cluster-maint |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-12-01 07:28:38 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
I was talking to klaus about this yesterday, its something to do with the remote connection resource. If you 'killall -TRAP crmd' a few times you see thousands of copies of: trace May 04 03:58:04 mainloop_gio_callback(673):0: New message from remote-lrmd-pcmk-2:1111[0x7f9def161c60] 1 trace May 04 03:58:04 lrmd_tls_dispatch(366):0: tls dispatch triggered after disconnect And if you think I'm exaggerating: [root@pcmk-2 /]# qb-blackbox /var/lib/pacemaker/blackbox/crmd-349.2 | grep "trace May 04 03:58:04 mainloop_gio_callback" | wc -l 4317 The message in particular suggests we're not correctly cleaning up dead connections. Oh and the other symptom is that all IPC to the crmd (eg. subsequent cleanup calls) fails. This is unlikely to be solved in the 7.5 timeframe Whoops, picked wrong drop-down Will consider for RHEL 8 After evaluating this issue, there are no plans to address it further or fix it in an upcoming release. Therefore, it is being closed. If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened. |
Description of problem: Running "crm_resource --resource <bundle resource> --cleanup" may result in the crmd process on the DC node taking 100 % of a CPU. Version-Release number of selected component (if applicable): pacemaker-1.1.16-8.el7.x86_64 How reproducible: easily, most of the time Steps to Reproduce: 1. create a bundle 2. run "crm_resource --resource <bundle resource> --cleanup" 3. it may be neede to run the command several Actual results: # crm_resource --resource http-bundle --cleanup Cleaning up http-bundle-docker-0 on rh73-node1, removing fail-count-http-bundle-docker-0 Cleaning up http-bundle-docker-0 on rh73-node2, removing fail-count-http-bundle-docker-0 Cleaning up http-bundle-ip-192.168.122.250 on rh73-node1, removing fail-count-http-bundle-ip-192.168.122.250 Cleaning up http-bundle-ip-192.168.122.250 on rh73-node2, removing fail-count-http-bundle-ip-192.168.122.250 Cleaning up http-bundle-0 on rh73-node1, removing fail-count-http-bundle-0 Cleaning up http-bundle-0 on rh73-node2, removing fail-count-http-bundle-0 Cleaning up http-bundle-docker-1 on rh73-node1, removing fail-count-http-bundle-docker-1 Cleaning up http-bundle-docker-1 on rh73-node2, removing fail-count-http-bundle-docker-1 Cleaning up http-bundle-ip-192.168.122.251 on rh73-node1, removing fail-count-http-bundle-ip-192.168.122.251 Cleaning up http-bundle-ip-192.168.122.251 on rh73-node2, removing fail-count-http-bundle-ip-192.168.122.251 Cleaning up http-bundle-1 on rh73-node1, removing fail-count-http-bundle-1 Cleaning up http-bundle-1 on rh73-node2, removing fail-count-http-bundle-1 Cleaning up dummy1:0 on rh73-node1, removing fail-count-dummy1 Cleaning up dummy1:0 on rh73-node2, removing fail-count-dummy1 Waiting for 14 replies from the CRMd.............. OK Then crmd takes 100 % of a CPU. Sometimes the "OK" at the last line is missing. Expected results: crmd does not take 100 % of a CPU. Additional info: Bundle configuration: <bundle id="http-bundle"> <docker image="pcmktest:http" options="--log-driver=journald" replicas="2"/> <network host-netmask="24" host-interface="ens3" ip-range-start="192.168.122.250"> <port-mapping port="80" id="http-bundle-port-map-80"/> </network> <storage> <storage-mapping source-dir-root="/root/docker/www" target-dir="/var/www/html" options="rw" id="http-bundle-storage-map"/> <storage-mapping source-dir-root="/root/docker/logs" target-dir="/etc/httpd/logs" options="rw" id="http-bundle-storage-map-1"/> </storage> <primitive class="ocf" id="dummy1" provider="pacemaker" type="Stateful"> <operations> <op id="dummy1-monitor-interval-10" interval="10" name="monitor" role="Master" timeout="20"/> <op id="dummy1-monitor-interval-11" interval="11" name="monitor" role="Slave" timeout="20"/> <op id="dummy1-start-interval-0s" interval="0s" name="start" timeout="20"/> <op id="dummy1-stop-interval-0s" interval="0s" name="stop" timeout="20"/> </operations> </primitive> </bundle>