Bug 1505540

Summary: fence_vmware_soap requires --force when configuring STONITH
Product: Red Hat Enterprise Linux 7 Reporter: Ryan <rblough>
Component: fence-agentsAssignee: Oyvind Albrigtsen <oalbrigt>
Status: CLOSED DUPLICATE QA Contact: cluster-qe <cluster-qe>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.4CC: cluster-maint, tojeline
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-10-24 15:04:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ryan 2017-10-23 20:13:11 UTC
Description of problem:

When configuring the fence_vmware_soap STONITH device in a Pacemaker cluster on RHEL 7.4, the pcs command threw an error that said we required different parameters.

Version-Release number of selected component (if applicable):

fence-agents-all-4.0.11-66.el7.x86_64
fence-agents-vmware-soap-4.0.11-66.el7.x86_64

How reproducible:
On the customer's system, consistently. It has been observed in two cases thus far.

Steps to Reproduce:
1. Test the fencing agent and then configure according to the customer-facing guidance: https://access.redhat.com/solutions/917813

2. Get the 'missing required option(s)' error

3. The same command with the --force option succeeds

4. The stonith device operates correctly

Actual results:

[root@eawvorsoaap003s ~]# pcs stonith create vmfence fence_vmware_soap pcmk_host_map="node1:EAWVORSOAAP003S-d93885c3-c679-4326-a887-2e9d085a761c;node2:EAWVORSOAAP004S-354828c2-ca43-4148-b206-55f00a63db6c" ipaddr=172.22.124.104 ssl_insecure=1 login=<login> passwd_script="/etc/corosync/fence/decrypt-fence-key" action=reboot delay=10
Error: missing required option(s): 'plug, username, ip' for resource type: stonith:fence_vmware_soap (use --force to override)

The password script was validated through visual inspection and manual testing.

Expected results:

The command succeeds without errors.

Additional info:

Comment 2 Ryan 2017-10-23 21:36:59 UTC
Matching the packages shown in the customer's sosreport, here is the output from pcs for the fence_vmware_soap agent:

[root@rhel73_generic ~]# pcs stonith describe fence_vmware_soap
fence_vmware_soap - Fence agent for VMWare over SOAP API

fence_vmware_soap is an I/O Fencing agent which can be used with the virtual machines managed by VMWare products that have SOAP API v4.1+. 
.P
Name of virtual machine (-n / port) has to be used in inventory path format (e.g. /datacenter/vm/Discovered virtual machine/myMachine). In the cases when name of yours VM is unique you can use it instead. Alternatively you can always use UUID to access virtual machine.

Resource options:
  ipport: TCP/UDP port to use for connection with device
  notls: Disable TLS negotiation, force SSL 3.0
  ssl_secure: SSL connection with verifying fence device's certificate
  port (required): Physical plug number, name of virtual machine or UUID
  inet6_only: Forces agent to use IPv6 addresses only
  ipaddr (required): IP Address or Hostname
  inet4_only: Forces agent to use IPv4 addresses only
  passwd_script: Script to retrieve password
  passwd: Login password or passphrase
  ssl: SSL connection
  ssl_insecure: SSL connection without verifying fence device's certificate
  action: Fencing Action WARNING: specifying 'action' is deprecated and not necessary with current Pacemaker versions
  login (required): Login Name
  plug (required): Physical plug number, name of virtual machine or UUID
  username (required): Login Name
  ip (required): IP Address or Hostname
  password: Login password or passphrase
  password_script: Script to retrieve password
  debug_file: Write debug information to given file
  separator: Separator for CSV created by operation list
  power_wait: Wait X seconds after issuing ON/OFF
  login_timeout: Wait X seconds for cmd prompt after login
  power_timeout: Test X seconds for status change after ON/OFF
  delay: Wait X seconds before fencing is started
  shell_timeout: Wait X seconds for cmd prompt after issuing command
  retry_on: Count of attempts to retry power on
  priority: The priority of the stonith resource. Devices are tried in order of highest priority to lowest.
  pcmk_host_map: A mapping of host names to ports numbers for devices that do not support host names. Eg. node1:1;node2:2,3 would tell the cluster to use port 1 for node1 and ports 2
                 and 3 for node2
  pcmk_host_list: A list of machines controlled by this device (Optional unless pcmk_host_check=static-list).
  pcmk_host_check: How to determine which machines are controlled by the device. Allowed values: dynamic-list (query the device), static-list (check the pcmk_host_list attribute), none
                   (assume every device can fence every machine)
  pcmk_delay_max: Enable random delay for stonith actions and specify the maximum of random delay This prevents double fencing when using slow devices such as sbd. Use this to enable
                  random delay for stonith actions and specify the maximum of random delay.
  pcmk_action_limit: The maximum number of actions can be performed in parallel on this device Pengine property concurrent-fencing=true needs to be configured first. Then use this to
                     specify the maximum number of actions can be performed in parallel on this device. -1 is unlimited.


Comparing to the parameters listed as required in the fence_vmware_soap metadata:

[root@rhel73_generic ~]# fence_vmware_soap -o metadata | grep 'required="1"'
	<parameter name="port" unique="0" required="1" deprecated="1">
	<parameter name="ipaddr" unique="0" required="1" deprecated="1">
	<parameter name="action" unique="0" required="1">
	<parameter name="login" unique="0" required="1" deprecated="1">
	<parameter name="plug" unique="0" required="1" obsoletes="port">
	<parameter name="username" unique="0" required="1" obsoletes="login">
	<parameter name="ip" unique="0" required="1" obsoletes="ipaddr">

So port, ipaddr, and login have been deprecated by plug, ip, and username respectively, but the parameter validation appears to be requiring all six entries. Naively I would expect an either/or logic, like so:

port/plug

ipaddr/ip

login/username

Or I would expect a rejection of the deprecated parameters with a message that they have been replaced by the new ones. However, when troubleshooting with the customer we saw that if we switched to the new parameter it would still throw an error requiring the old one.

I am trying to replicate the problem on my test machine.

Comment 3 Tomas Jelinek 2017-10-24 07:59:02 UTC
I think this has been fixed in pcs-0.9.157-1.el7, see bz1434972 for details.

[root@rh74-node1:~]# rpm -q pcs
pcs-0.9.158-6.el7.x86_64
[root@rh74-node1:~]# pcs stonith create vmfence fence_vmware_soap pcmk_host_map="node1:EAWVORSOAAP003S-d93885c3-c679-4326-a887-2e9d085a761c;node2:EAWVORSOAAP004S-354828c2-ca43-4148-b206-55f00a63db6c" ipaddr=172.22.124.104 ssl_insecure=1 login=login passwd_script="/etc/corosync/fence/decrypt-fence-key" action=reboot delay=10
[root@rh74-node1:~]# echo $?
0
[root@rh74-node1:~]# pcs stonith show vmfence
 Resource: vmfence (class=stonith type=fence_vmware_soap)
  Attributes: action=reboot delay=10 ipaddr=172.22.124.104 login=login passwd_script=/etc/corosync/fence/decrypt-fence-key pcmk_host_map=node1:EAWVORSOAAP003S-d93885c3-c679-4326-a887-2e9d085a761c;node2:EAWVORSOAAP004S-354828c2-ca43-4148-b206-55f00a63db6c ssl_insecure=1
  Operations: monitor interval=60s (vmfence-monitor-interval-60s)


[root@rh74-node1:~]# pcs stonith describe fence_vmware_soap
fence_vmware_soap - Fence agent for VMWare over SOAP API

fence_vmware_soap is an I/O Fencing agent which can be used with the virtual machines managed by VMWare products that have SOAP API v4.1+. 
.P
Name of virtual machine (-n / port) has to be used in inventory path format (e.g. /datacenter/vm/Discovered virtual machine/myMachine). In the cases when name of yours VM is unique you can use it instead. Alternatively you can always use UUID to access virtual machine.

Stonith options:
  ipport: TCP/UDP port to use for connection with device
  notls: Disable TLS negotiation, force SSL 3.0
  ssl_secure: SSL connection with verifying fence device's certificate
  port: Physical plug number, name of virtual machine or UUID
  inet6_only: Forces agent to use IPv6 addresses only
  ipaddr (required): IP Address or Hostname
  inet4_only: Forces agent to use IPv4 addresses only
  passwd_script: Script to retrieve password
  passwd: Login password or passphrase
  ssl: SSL connection
  ssl_insecure: SSL connection without verifying fence device's certificate
  action: Fencing Action WARNING: specifying 'action' is deprecated and not necessary with current Pacemaker versions.
  login (required): Login Name
  separator: Separator for CSV created by operation list
  power_wait: Wait X seconds after issuing ON/OFF
  login_timeout: Wait X seconds for cmd prompt after login
  power_timeout: Test X seconds for status change after ON/OFF
  delay: Wait X seconds before fencing is started
  shell_timeout: Wait X seconds for cmd prompt after issuing command
  retry_on: Count of attempts to retry power on
  priority: The priority of the stonith resource. Devices are tried in order of highest priority to lowest.
  pcmk_host_map: A mapping of host names to ports numbers for devices that do not support host names. Eg. node1:1;node2:2,3 would tell the cluster to use port 1 for node1 and ports 2 and 3 for node2
  pcmk_host_list: A list of machines controlled by this device (Optional unless pcmk_host_check=static-list).
  pcmk_host_check: How to determine which machines are controlled by the device. Allowed values: dynamic-list (query the device), static-list (check the pcmk_host_list attribute), none (assume every device can fence every machine)
  pcmk_delay_max: Enable a random delay for stonith actions and specify the maximum of random delay. This prevents double fencing when using slow devices such as sbd. Use this to enable a random delay for stonith actions. The overall delay is derived from this random
                  delay value adding a static delay so that the sum is kept below the maximum delay.
  pcmk_delay_base: Enable a base delay for stonith actions and specify base delay value. This prevents double fencing when different delays are configured on the nodes. Use this to enable a static delay for stonith actions. The overall delay is derived from a random delay
                   value adding this static delay so that the sum is kept below the maximum delay.
  pcmk_action_limit: The maximum number of actions can be performed in parallel on this device Pengine property concurrent-fencing=true needs to be configured first. Then use this to specify the maximum number of actions can be performed in parallel on this device. -1 is
                     unlimited.

Default operations:
  monitor: interval=60s
[root@rh74-node1:~]# fence_vmware_soap -o metadata | grep 'required="1"'
        <parameter name="port" unique="0" required="1" deprecated="1">
        <parameter name="ipaddr" unique="0" required="1" deprecated="1">
        <parameter name="action" unique="0" required="1">
        <parameter name="login" unique="0" required="1" deprecated="1">
        <parameter name="plug" unique="0" required="1" obsoletes="port">
        <parameter name="username" unique="0" required="1" obsoletes="login">
        <parameter name="ip" unique="0" required="1" obsoletes="ipaddr">

Comment 4 Ryan 2017-10-24 15:03:14 UTC
Testing:

[root@cs-rh7-7 ~]# rpm -q pcs fence-agents-common fence-agents-vmware-soap
pcs-0.9.152-10.el7_3.3.x86_64
fence-agents-common-4.0.11-66.el7_4.3.x86_64
fence-agents-vmware-soap-4.0.11-66.el7_4.3.x86_64

[root@cs-rh7-7 ~]# pcs stonith create test fence_vmware_soap
Error: missing required option(s): 'ipaddr, login, plug, username, ip' for resource type: stonith:fence_vmware_soap (use --force to override)

[root@cs-rh7-7 ~]# pcs stonith create test fence_vmware_soap ipaddr=test login=test plug=test
Error: missing required option(s): 'username, ip' for resource type: stonith:fence_vmware_soap (use --force to override)

[root@cs-rh7-7 ~]# pcs stonith create test fence_vmware_soap ip=test username=test plug=test
Error: missing required option(s): 'ipaddr, login' for resource type: stonith:fence_vmware_soap (use --force to override)


Then applying the fix from the other bug:

[root@cs-rh7-7 ~]# rpm -q pcs
pcs-0.9.158-6.el7.x86_64

[root@cs-rh7-7 ~]# pcs stonith create test fence_vmware_soap ip=test username=test plug=test
Error: invalid resource options: 'ip', 'plug', 'username', allowed options are: action, delay, inet4_only, inet6_only, ipaddr, ipport, login, login_timeout, notls, passwd, passwd_script, pcmk_action_limit, pcmk_delay_max, pcmk_host_argument, pcmk_host_check, pcmk_host_list, pcmk_host_map, pcmk_list_action, pcmk_list_retries, pcmk_list_timeout, pcmk_monitor_action, pcmk_monitor_retries, pcmk_monitor_timeout, pcmk_off_action, pcmk_off_retries, pcmk_off_timeout, pcmk_reboot_action, pcmk_reboot_retries, pcmk_reboot_timeout, pcmk_status_action, pcmk_status_retries, pcmk_status_timeout, port, power_timeout, power_wait, priority, retry_on, separator, shell_timeout, ssl, ssl_insecure, ssl_secure, use --force to override
Error: required resource options 'ipaddr', 'login' are missing, use --force to override

[root@cs-rh7-7 ~]# pcs stonith create test fence_vmware_soap ipaddr=test login=test

[root@cs-rh7-7 ~]# echo $?
0

[root@cs-rh7-7 ~]# pcs stonith show test
 Resource: test (class=stonith type=fence_vmware_soap)
  Attributes: ipaddr=test login=test
  Operations: monitor interval=60s (test-monitor-interval-60s)

Fix confirmed! I will mark this as a duplicate.

Comment 5 Ryan 2017-10-24 15:04:22 UTC

*** This bug has been marked as a duplicate of bug 1434972 ***