Created attachment 1820042 [details]
normal log file

Hi,

What exactly is partial in the bad output? Are there resources or clone instances missing that should be active?

(In reply to Ken Gaillot from comment #2)
> Hi,
> 
> What exactly is partial in the bad output? Are there resources or clone
> instances missing that should be active?

please check attachment which shows the "good" and "bad" report. the bad report is miss half lines of resource status.

(In reply to tbsky from comment #3)
> (In reply to Ken Gaillot from comment #2)
> > Hi,
> > 
> > What exactly is partial in the bad output? Are there resources or clone
> > instances missing that should be active?
> 
> please check attachment which shows the "good" and "bad" report. the bad
> report is miss half lines of resource status.

Actually you can see that in the bad report, the resources which are running at "love-2-test.example.com" are gone. only resources which are running at "love-1-test.example.com" left.

Would you be able to attach /var/log/pacemaker/pacemaker.log from both nodes from the time of the failure (Sep 2, which might be rotated)?

Created attachment 1821932 [details]
server 1 log

Created attachment 1821933 [details]
server 2(dc) log

Hi:
   attached pacemaker logs. hope they can reveal something.

Thanks for the logs.

When a node joins the cluster's controller group, the DC clears its previous resource history and adds the current history state as given in the join message. The problem is that the DC does this in two separate CIB requests, and if the crm_mon happens to run between the two, it will see no resource history for the joining node.

The solution will be to consolidate these into a single CIB request (which will require coding some new capabilities in the CIB). We can use this bz to track that fix.

Separately, in the attached logs, your cluster is more likely to hit this because the nodes are continuously starting new DC elections due to the CIB being replaced. It appears something is repeatedly changing the configuration during this time with the cibadmin tool (or a wrapper such as pcs). Examples of the changes:

    Sep 02 20:41:16 love-1-test.example.com pacemaker-based     [2781] (cib_perform_op)     info: -- /cib/configuration/constraints/rsc_location[@id='location-kvm-11-love-2-test.example.com-99']
    ...
    Sep 02 20:41:17 love-1-test.example.com pacemaker-based     [2781] (cib_perform_op)     info: ++ /cib/configuration/constraints:  <rsc_location id="location-kvm-11-love-1-test.example.com-99" node="love-1-test.example.com" rsc="kvm-11" score="99"/>
    ...
    Sep 02 20:42:05 love-1-test.example.com pacemaker-based     [2781] (cib_perform_op)     info: +  /cib/configuration/resources/clone[@id='kvm-11-drbd-clone']/meta_attributes[@id='kvm-11-drbd-clone-meta_attributes']/nvpair[@id='kvm-11-drbd-clone-meta_attributes-promoted-max']:  @value=2
    ...
    Sep 02 20:42:08 love-1-test.example.com pacemaker-based     [2781] (cib_perform_op)     info: +  /cib/configuration/resources/primitive[@id='kvm-11']/meta_attributes[@id='kvm-11-meta_attributes']/nvpair[@id='kvm-11-meta_attributes-allow-migrate']:  @value=true

and so on. If the CIB is simply modified (cibadmin --modify/--delete), no election is needed, but if the entire CIB is replaced (--replace, which I believe pcs uses), the nodes may have changed, so a new DC election is needed. The new election triggers the erasing and resetting of the resource history. It might be possible for us to avoid calling a new election if the nodes have not changed, but I'm not sure that's doable in a reliable and quick way.

The first thing I'd do is identify what's changing the configuration. Then there may be an obvious way to mitigate the issue.

Thanks for the detailed explain. 

Indeed I use pcs to manage the whole cluster, but sometimes something may go wrong and not repeatable. so I decide to write a testing script which create/delete/migrate drbd/vm resources(mostly with pcs command). the testing script result is highly unstable, the cluster sometimes reports failed resource without any reason. I don't know what's wrong since there are many other elements involved like libvirtd,drbd and their resource agents. the only thing I am sure is that "crm_mon -1" should be related to pacemaker so I report it.

now I doubt all these come from the same source as you described. 

among all these situation, "vm migration failed" annoys me most. I can see migration succeed logs in the first server, but suddenly peer server decide the resource "cannot run anywhere" and the vm resource need to destroy at both servers. the behavior is not repeatable every time, so I spent weeks trying to find out what's the racing. I will try to modify the test script to see if I can get stable result.

thanks again for your kindly help!

(In reply to Ken Gaillot from comment #9)
> Separately, in the attached logs, your cluster is more likely to hit this
> because the nodes are continuously starting new DC elections due to the CIB
> being replaced. It appears something is repeatedly changing the
> configuration during this time with the cibadmin tool (or a wrapper such as
> pcs). Examples of the changes:
> ...
> 
> and so on. If the CIB is simply modified (cibadmin --modify/--delete), no
> election is needed, but if the entire CIB is replaced (--replace, which I
> believe pcs uses), the nodes may have changed, so a new DC election is
> needed. The new election triggers the erasing and resetting of the resource
> history. It might be possible for us to avoid calling a new election if the
> nodes have not changed, but I'm not sure that's doable in a reliable and
> quick way.

This part might be fixed upstream already by f522821.
  - Mid: based controld: Suppresses unnecessary Election execution. (https://github.com/ClusterLabs/pacemaker/pull/2547/commits/f522821)

(In reply to Reid Wahl from comment #11)
> This part might be fixed upstream already by f522821.
>   - Mid: based controld: Suppresses unnecessary Election execution.
> (https://github.com/ClusterLabs/pacemaker/pull/2547/commits/f522821)

Depends on the specifics. `pcs` often replaces only the configuration but will replace the entire CIB if `pcs cluster cib-push` is run without `--config` or `--scope configuration`.

After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.

Fixed in upstream main branch as of commit 5e3b3d142

After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.

Reopening because this is fixed as mentioned in comment 16.

(In reply to Reid Wahl from comment #20)
> Reopening because this is fixed as mentioned in comment 16.

5e3b3d142 introduced a regression though, so a fix would need the commits in https://github.com/ClusterLabs/pacemaker/pull/3180/commits as well. At least through 44c51814, but we should grab them all to be safe.

Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.

This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.