Bug 1230368

Note that this situation was created just by pcs commands:

# pcs cluster setup ...
# pcs -f cib.xml ...; pcs -f cib.xml ...; ...
# pcs cluster cib-push cib.xml --config

so a user carelessness is not be be blamed here.

* so a user's carelessness is not to be blamed here


Note that underlying cibadmin command does support also --xpath option
and it would be helpfull:

1/ add support for xpath passthrough (e.g., if scope starts with
   a slash, treat it as an XPath expression)

2/ --config option was made an alias to
>  scope=/cib/config/*[name() != 'nodes']
   so the nodes section is not included (hence overwritten)

re [comment 2]:

Apparently, the clarity is lacking behind if it's not clear that cluster
was indeed started, hence correcting the first command to:

# pcs cluster setup --start ...

Note that in the situation per [comment 0], crm_node still produces
expected results (confirming that Pacemaker is perfectly aware of
the nodes, unlike pcs):

# crm_node -l
> virt-092 virt-094 virt-097

"pcs cluster cib-push doesn't allow to utilize XPath filter provided by
underlying cibadmin" issue covered in [comment 2] and [comment 3] split
off as a separate [bug 1230908].

Proper reproducer:

pcs cluster setup --name test rh67-node1 rh67-node2 --start && \
for i in {1..3}; do pcs -f aaa.xml resource create dummy${i} Dummy; done && \
pcs cluster cib-push aaa.xml && \
echo '===== crm_node =====' && \
crm_node -l && \
echo '===== crm_mon =====' && \
crm_mon -1 && \
echo '===== pcs status =====' && \
pcs status

[root@rh67-node1:~]# pcs cluster setup --name test rh67-node1 rh67-node2 --start && \
> for i in {1..3}; do pcs -f aaa.xml resource create dummy${i} Dummy; done && \
> pcs cluster cib-push aaa.xml && \
> echo '===== crm_node =====' && \
> crm_node -l && \
> echo '===== crm_mon =====' && \
> crm_mon -1 && \
> echo '===== pcs status =====' && \
> pcs status
rh67-node1: Updated cluster.conf...
rh67-node2: Updated cluster.conf...
Starting cluster on nodes: rh67-node1, rh67-node2...
rh67-node2: Starting Cluster...
rh67-node1: Starting Cluster...
CIB updated
===== crm_node =====
rh67-node1 rh67-node2 
===== crm_mon =====
Last updated: Wed Sep  9 10:27:26 2015
Last change: Wed Sep  9 10:27:26 2015
Current DC: NONE
0 Nodes configured
3 Resources configured



===== pcs status =====
Cluster name: test
WARNING: no stonith devices and stonith-enabled is not false
Last updated: Wed Sep  9 10:27:26 2015
Last change: Wed Sep  9 10:27:26 2015
Current DC: NONE
0 Nodes configured
3 Resources configured



Full list of resources:

 dummy1 (ocf::heartbeat:Dummy): Stopped 
 dummy2 (ocf::heartbeat:Dummy): Stopped 
 dummy3 (ocf::heartbeat:Dummy): Stopped 



As expected it's actually crm_mon output where the nodes are missing. After a while the issue is fixed:

[root@rh67-node1:~]# crm_mon -1
Last updated: Wed Sep  9 10:28:06 2015
Last change: Wed Sep  9 10:27:41 2015
Stack: cman
Current DC: rh67-node2 - partition with quorum
Version: 1.1.11-97629de
2 Nodes configured
3 Resources configured


Online: [ rh67-node1 rh67-node2 ]



We need to investigate this further.

If the CIB from the cluster is used instead of an empty one, everything works:

[root@rh67-node1:~]# pcs cluster setup --name test rh67-node1 rh67-node2 --start && \
> pcs cluster cib > aaa.xml && \
> for i in {1..3}; do pcs -f aaa.xml resource create dummy${i} Dummy; done && \
> pcs cluster cib-push aaa.xml && \
> echo '===== crm_node =====' && \
> crm_node -l && \
> echo '===== crm_mon =====' && \
> crm_mon -1 && \
rh67-node1: Updated cluster.conf...
rh67-node2: Updated cluster.conf...
Starting cluster on nodes: rh67-node1, rh67-node2...
rh67-node2: Starting Cluster...
rh67-node1: Starting Cluster...
CIB updated
===== crm_node =====
rh67-node1 rh67-node2 
===== crm_mon =====
Last updated: Wed Sep  9 10:40:07 2015
Last change: Wed Sep  9 10:40:07 2015
Current DC: NONE
2 Nodes configured
3 Resources configured


Node rh67-node1: UNCLEAN (offline)
Node rh67-node2: UNCLEAN (offline)


So the cause of the reported issue is the nodes section of the CIB has been overwritten by an empty list of nodes from an empty CIB.

The cause of the reported issue is the nodes section of the CIB has been overwritten by an empty list of nodes from an empty CIB. This is perfectly OK as it is what user wanted to do by pushing an empty node list.

However help text/documentation of the 'pcs cluster cib-push' command could be improved e.g. like this:
Push the raw xml from <filename> to the CIB (Cluster Information Base).  You can obtain the CIB by running the 'pcs cluster cib' command, which is recommended first step when you want to perform desired modifications (pcs -f <command>) for the one-off push.
Be aware the selected scope of the CIB will be overwritten by the current content of the specified file.

The original gear exercising the recipe per [comment 2] was revisited
to fix the assumptions behind how "pcs cluster cib-push" works, i.e.,
along the lines of [comment 9] -- one should not start with temporary
CIB file not preinitiated with the currently used content so as to
prevent from dropping existing bits undesirably, but start with them
right away instead: https://pagure.io/clufter/0ebae33

Having the better documentation beforehand would definitely help.

Created attachment 1072155 [details]
proposed fix

After Fix:
[root@rh67-node1:~]# rpm -q pcs
pcs-0.9.145-1.el6.x86_64
Help and man page modified as described in comment 9.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0739.html