Description of problem:

When a remote console has a special character like $ somewhere in the middle, the configuration of the corresponding stonith device gets a broken password, which causes the fence device to remain in "stopped" status. This may also be a problem with other characters, but I didn't try them all.

Version-Release number of selected component (if applicable):
Tested in 16.1

How reproducible:
Always reproducible when the console password includes $

Steps to Reproduce:
1. Create nodes.yaml where at least one remote console has its password include $:

~~~
(undercloud) [stack.lab ~]$ egrep "name|pm_type|pm_user|pm_password" nodes.yaml |head -4
  - name: "overcloud-controller-0.keller1618.lab"
    pm_type: "ipmi"
    pm_user: "admin"
    pm_password: "some$tring"
~~~

2. Generate fencing.yaml:

~~~
(undercloud) [stack.lab ~]$ openstack overcloud generate fencing --output /home/stack/templates/fencing.yaml /home/stack/nodes.yaml
(undercloud) [stack.lab ~]$ head -13 /home/stack/templates/fencing.yaml
parameter_defaults:
  EnableFencing: true
  FencingConfig:
    devices:
    - agent: fence_ipmilan
      host_mac: 52:54:00:db:a1:22
      params:
        ipaddr: 192.168.122.1
        ipport: '16021'
        lanplus: true
        login: admin
        passwd: some$tring
        pcmk_host_list: overcloud-controller-0
~~~

3. Add nodes, tag profiles, introspect, etc.

4. Deploy overcloud

Actual results:

Stonith device corresponding to overcloud-controller-0 is in Stopped state (despite overcloud deploy completing successfully):

~~~
[root@overcloud-controller-0 ~]# pcs stonith show
Warning: This command is deprecated and will be removed. Please use 'pcs stonith status' instead.
  * stonith-fence_ipmilan-525400cd455b	(stonith:fence_ipmilan):	Started overcloud-controller-0
  * stonith-fence_ipmilan-525400210a64	(stonith:fence_ipmilan):	Started overcloud-controller-2
  * stonith-fence_ipmilan-525400dba122	(stonith:fence_ipmilan):	Stopped
 Target: overcloud-controller-0
   Level 1 - stonith-fence_ipmilan-525400dba122
 Target: overcloud-controller-1
   Level 1 - stonith-fence_ipmilan-525400210a64
 Target: overcloud-controller-2
   Level 1 - stonith-fence_ipmilan-525400cd455b
~~~

Stonith configuration suggests that the 2nd portion of the string was evaluated to null:

[root@overcloud-controller-0 ~]# pcs stonith show stonith-fence_ipmilan-525400dba122 | grep passwd
  Attributes: ipaddr=192.168.122.1 ipport=16021 lanplus=true login=admin passwd=some pcmk_host_list=overcloud-controller-0
[root@overcloud-controller-0 ~]# 
~~~

Expected results:

Stonith devices should receive passwords matching the information in fencing.yaml


Additional info:

The issue seems to be related to the stonith manifests of puppet-pacemaker [0] (customer reported it when using ilo driver, but I could easily reproduce when using plain ipmi). Other tasks that require console, such as introspection, complete successfully. Also the password is correct in Ansible:

~~~
(undercloud) [stack.lab ~]$ sudo grep "passwd: some" /var/lib/mistral/overcloud/group_vars/Controller 
        passwd: some$tring
~~~

If I patch /etc/puppet/modules/pacemaker/manifests/stonith/fence_ipmilan.pp in all controllers like this before stonith devices are created, then they are created successfully and the ipmi device is started:

~~~
[root@overcloud-controller-2 ~]# cd /etc/puppet/modules/pacemaker/
[root@overcloud-controller-2 pacemaker]# git diff
diff --git a/manifests/stonith/fence_ipmilan.pp b/manifests/stonith/fence_ipmilan.pp
index 2b2babc..d622c56 100644
--- a/manifests/stonith/fence_ipmilan.pp
+++ b/manifests/stonith/fence_ipmilan.pp
@@ -148,7 +148,7 @@ define pacemaker::stonith::fence_ipmilan (
   }
   $passwd_chunk = $passwd ? {
     undef   => '',
-    default => "passwd=\"${passwd}\"",
+    default => "passwd=\'${passwd}\'",
   }
   $passwd_script_chunk = $passwd_script ? {
     undef   => '',
[root@overcloud-controller-2 pacemaker]# 
[root@overcloud-controller-2 pacemaker]# pcs stonith
  * stonith-fence_ipmilan-525400cd455b	(stonith:fence_ipmilan):	Started overcloud-controller-0
  * stonith-fence_ipmilan-525400210a64	(stonith:fence_ipmilan):	Started overcloud-controller-2
  * stonith-fence_ipmilan-525400dba122	(stonith:fence_ipmilan):	Started overcloud-controller-1
 Target: overcloud-controller-0
   Level 1 - stonith-fence_ipmilan-525400dba122
 Target: overcloud-controller-1
   Level 1 - stonith-fence_ipmilan-525400210a64
 Target: overcloud-controller-2
   Level 1 - stonith-fence_ipmilan-525400cd455b
[root@overcloud-controller-2 pacemaker]# 
[root@overcloud-controller-2 pacemaker]# pcs stonith show stonith-fence_ipmilan-525400dba122 | grep passwd
  Attributes: ipaddr=192.168.122.1 ipport=16021 lanplus=true login=admin passwd=some$tring pcmk_host_list=overcloud-controller-0
[root@overcloud-controller-2 pacemaker]# 
~~~

Workarounds attempted:

1. Enclosing password between single quotes in nodes.yaml: This does not work, as the quotes are removed when fencing.yaml is generated:

~~~
(undercloud) [stack.lab ~]$ egrep "name|pm_type|pm_user|pm_password" nodes.yaml |head -4
  - name: "overcloud-controller-0.keller1618.lab"
    pm_type: "ipmi"
    pm_user: "admin"
    pm_password: 'some$tring'
(undercloud) [stack.lab ~]$ openstack overcloud generate fencing /home/stack/nodes.yaml | head -13
parameter_defaults:
  EnableFencing: true
  FencingConfig:
    devices:
    - agent: fence_ipmilan
      host_mac: 52:54:00:db:a1:22
      params:
        ipaddr: 192.168.122.1
        ipport: '16021'
        lanplus: true
        login: admin
        passwd: some$tring
        pcmk_host_list: overcloud-controller-0
~~~

2. Manually enclosing password between single quotes in fencing.yaml: This does not work, as the manifest in the puppet-pacemaker module still evaluates the string

3. Manually adding backslash to the password in fencing.yaml: This works, but I would say this is a workaround since fencing.yaml is supposed to be generated automatically and not manually crafted (escaping $ in nodes.yaml does not work as it breaks the fencing.yaml generation workflow).


[0] https://opendev.org/openstack/puppet-pacemaker/src/branch/master/manifests/stonith/fence_ipmilan.pp#L151