Created attachment 1820041 [details] bad situation log file Description of problem: I have scripts which use 'crm_non -1'. and sometimes the script failed. after debugging I found sometimes 'crm_mon -1' didn't report full information Version-Release number of selected component (if applicable): pacemaker-cli-2.1.0-8.el8.x86_64 How reproducible: sometimes Steps to Reproduce: 1. do some change to cluster configuration 2. run 'crm_mon -1' to get the cluster status 3. Actual results: sometimes the report is not full, only part of information Expected results: full correct information Additional info: I notice when the report time of "Last updated:" and "Last change:" are the same or very close, then it may trigger the problem. please check attachment which shows the "good" and "bad" report.
Created attachment 1820042 [details] normal log file
Hi, What exactly is partial in the bad output? Are there resources or clone instances missing that should be active?
(In reply to Ken Gaillot from comment #2) > Hi, > > What exactly is partial in the bad output? Are there resources or clone > instances missing that should be active? please check attachment which shows the "good" and "bad" report. the bad report is miss half lines of resource status.
(In reply to tbsky from comment #3) > (In reply to Ken Gaillot from comment #2) > > Hi, > > > > What exactly is partial in the bad output? Are there resources or clone > > instances missing that should be active? > > please check attachment which shows the "good" and "bad" report. the bad > report is miss half lines of resource status. Actually you can see that in the bad report, the resources which are running at "love-2-test.example.com" are gone. only resources which are running at "love-1-test.example.com" left.
Would you be able to attach /var/log/pacemaker/pacemaker.log from both nodes from the time of the failure (Sep 2, which might be rotated)?
Created attachment 1821932 [details] server 1 log
Created attachment 1821933 [details] server 2(dc) log
Hi: attached pacemaker logs. hope they can reveal something.
Thanks for the logs. When a node joins the cluster's controller group, the DC clears its previous resource history and adds the current history state as given in the join message. The problem is that the DC does this in two separate CIB requests, and if the crm_mon happens to run between the two, it will see no resource history for the joining node. The solution will be to consolidate these into a single CIB request (which will require coding some new capabilities in the CIB). We can use this bz to track that fix. Separately, in the attached logs, your cluster is more likely to hit this because the nodes are continuously starting new DC elections due to the CIB being replaced. It appears something is repeatedly changing the configuration during this time with the cibadmin tool (or a wrapper such as pcs). Examples of the changes: Sep 02 20:41:16 love-1-test.example.com pacemaker-based [2781] (cib_perform_op) info: -- /cib/configuration/constraints/rsc_location[@id='location-kvm-11-love-2-test.example.com-99'] ... Sep 02 20:41:17 love-1-test.example.com pacemaker-based [2781] (cib_perform_op) info: ++ /cib/configuration/constraints: <rsc_location id="location-kvm-11-love-1-test.example.com-99" node="love-1-test.example.com" rsc="kvm-11" score="99"/> ... Sep 02 20:42:05 love-1-test.example.com pacemaker-based [2781] (cib_perform_op) info: + /cib/configuration/resources/clone[@id='kvm-11-drbd-clone']/meta_attributes[@id='kvm-11-drbd-clone-meta_attributes']/nvpair[@id='kvm-11-drbd-clone-meta_attributes-promoted-max']: @value=2 ... Sep 02 20:42:08 love-1-test.example.com pacemaker-based [2781] (cib_perform_op) info: + /cib/configuration/resources/primitive[@id='kvm-11']/meta_attributes[@id='kvm-11-meta_attributes']/nvpair[@id='kvm-11-meta_attributes-allow-migrate']: @value=true and so on. If the CIB is simply modified (cibadmin --modify/--delete), no election is needed, but if the entire CIB is replaced (--replace, which I believe pcs uses), the nodes may have changed, so a new DC election is needed. The new election triggers the erasing and resetting of the resource history. It might be possible for us to avoid calling a new election if the nodes have not changed, but I'm not sure that's doable in a reliable and quick way. The first thing I'd do is identify what's changing the configuration. Then there may be an obvious way to mitigate the issue.
Thanks for the detailed explain. Indeed I use pcs to manage the whole cluster, but sometimes something may go wrong and not repeatable. so I decide to write a testing script which create/delete/migrate drbd/vm resources(mostly with pcs command). the testing script result is highly unstable, the cluster sometimes reports failed resource without any reason. I don't know what's wrong since there are many other elements involved like libvirtd,drbd and their resource agents. the only thing I am sure is that "crm_mon -1" should be related to pacemaker so I report it. now I doubt all these come from the same source as you described. among all these situation, "vm migration failed" annoys me most. I can see migration succeed logs in the first server, but suddenly peer server decide the resource "cannot run anywhere" and the vm resource need to destroy at both servers. the behavior is not repeatable every time, so I spent weeks trying to find out what's the racing. I will try to modify the test script to see if I can get stable result. thanks again for your kindly help!
(In reply to Ken Gaillot from comment #9) > Separately, in the attached logs, your cluster is more likely to hit this > because the nodes are continuously starting new DC elections due to the CIB > being replaced. It appears something is repeatedly changing the > configuration during this time with the cibadmin tool (or a wrapper such as > pcs). Examples of the changes: > ... > > and so on. If the CIB is simply modified (cibadmin --modify/--delete), no > election is needed, but if the entire CIB is replaced (--replace, which I > believe pcs uses), the nodes may have changed, so a new DC election is > needed. The new election triggers the erasing and resetting of the resource > history. It might be possible for us to avoid calling a new election if the > nodes have not changed, but I'm not sure that's doable in a reliable and > quick way. This part might be fixed upstream already by f522821. - Mid: based controld: Suppresses unnecessary Election execution. (https://github.com/ClusterLabs/pacemaker/pull/2547/commits/f522821)
(In reply to Reid Wahl from comment #11) > This part might be fixed upstream already by f522821. > - Mid: based controld: Suppresses unnecessary Election execution. > (https://github.com/ClusterLabs/pacemaker/pull/2547/commits/f522821) Depends on the specifics. `pcs` often replaces only the configuration but will replace the entire CIB if `pcs cluster cib-push` is run without `--config` or `--scope configuration`.
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release. Therefore, it is being closed. If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.
Fixed in upstream main branch as of commit 5e3b3d142