Bug 1248149

Summary: 'Unable to connect/login to fencing device' when using secure cisco_ucs fencing
Product: Red Hat Enterprise Virtualization Manager Reporter: Robert McSwain <rmcswain>
Component: ovirt-engineAssignee: Oved Ourfali <oourfali>
Status: CLOSED DUPLICATE QA Contact:
Severity: high Docs Contact:
Priority: high    
Version: 3.5.3CC: colin, ecohen, emesika, lpeer, lsurette, mgoldboi, oourfali, rbalakri, Rhev-m-bugs, rmcswain, tdosek, yeylon
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard: infra
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-09-24 08:08:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Infra RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Robert McSwain 2015-07-29 17:38:49 UTC
I am having an issue configuring cisco_ucs fencing on a new RHEV cluster.  When I configure the Power Management settings and give it the proper credentials to connect to the UCS manager and the proper service profile name when I go to test the connection I get the status of "Test Succeeded, unknown".  When I attempt to reset the host inside the manager it is unsuccessful.  I'll attach output from the /var/log/ovirt-engine/engine.log that shows the test, an attempted restart and a screenshot of the configuration.

Along with the failures we get a message in the vdsm log on the hypervisor that a proxy request goes though that reports that the username / password is invalid however I am able to ssh into the UCS manager without any issues as that account.

Here is an example of the failure for it to connect:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Thread-27969::DEBUG::2015-07-22 16:24:55,756::stompReactor::163::yajsonrpc.StompServer::(send) Sending response
JsonRpc (StompReactor)::DEBUG::2015-07-22 16:24:55,757::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'>
JsonRpcServer::DEBUG::2015-07-22 16:24:55,758::__init__::530::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request
Thread-27970::DEBUG::2015-07-22 16:24:55,759::API::1209::vds::(fenceNode) fenceNode(addr=10.0.10.22,port=,agent=cisco_ucs,user=rhevm,passwd=XXXX,action=status,secure=False,options=port=hqvhost01
ssl=yes,policy=None)
Thread-27970::DEBUG::2015-07-22 16:24:55,759::utils::739::root::(execCmd) /usr/sbin/fence_cisco_ucs (cwd None)
Thread-26177::DEBUG::2015-07-22 16:24:55,821::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.0.12.90:_volume1_RHEVExport/df457e4d-536d-428b-87f1-79e3422325f5/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None)
Thread-26177::DEBUG::2015-07-22 16:24:55,834::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n340 bytes (340 B) copied, 0.00683977 s, 49.7 kB/s\n'; <rc> = 0
Thread-27970::DEBUG::2015-07-22 16:24:55,852::utils::759::root::(execCmd) FAILED: <err> = 'Unable to connect/login to fencing device\n\n\n'; <rc> = 1
Thread-27970::DEBUG::2015-07-22 16:24:55,852::API::1164::vds::(fence) rc 1 inp agent=fence_cisco_ucs
ipaddr=10.0.10.22
login=rhevm
action=status
passwd=XXXX
port=hqvhost01
ssl=yes out [] err ['Unable to connect/login to fencing device', '', '']
Thread-27970::DEBUG::2015-07-22 16:24:55,852::API::1235::vds::(fenceNode) rc 1 in agent=fence_cisco_ucs
ipaddr=10.0.10.22
login=rhevm
action=status
passwd=XXXX
port=hqvhost01
ssl=yes out [] err ['Unable to connect/login to fencing device', '', '']
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

We have determined that if we disable the secure option in Power Management for the host and disable the http to https redirection in the UCS manager we are able to successfully connect. Using the secure option would be the best since the UCS manager was redirecting all http requests over to https and we would like to switch that back if possible. 

Version:
Red Hat Enterprise Virtualization Hypervisor release 7.1 (20150603.0.el7ev)
rhevm-3.5.3.1-1.4.el6ev.noarch

Additional Information:

I will attach the output from running the fence_cisco_ucs script on one of the RHEV guests against the same UCS manager and you can see where http works while https does not.  The vdsm.logs on the hypervisors where the fencing commands get proxied to show the same unable to connect to fencing device message when we have the secure check box selected.

It works as designed with the secure option in power management disabled and http redirection to https disabled in the UCS manager but once we change it to secure it goes back to being unable to connect.

We have also discovered that you can pass --ssl-insecure to the script and it functions properly.  Unfortunately that is not an option in the RHEV-M power management option for the host.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[root@rheltest ~]# fence_cisco_ucs --ip=10.0.10.22 --username=rhevm --password=mrhev --plug=hqvhost04 --ssl --ssl-insecure -o status -vvv
<aaaLogin inName="rhevm" inPassword="mrhev" />

 <aaaLogin cookie="" response="yes" outCookie="1437593293/31565a68-084f-4029-b2ba-e48a80e98419" outRefreshPeriod="600" outPriv="admin,read-only" outDomains="" outChannel="noencssl" outEvtChannel="noencssl" outSessionId="web_723_B" outVersion="2.2(5a)" outName="rhevm"> </aaaLogin>

<configResolveDn cookie="1437593293/31565a68-084f-4029-b2ba-e48a80e98419" inHierarchical="false" dn="org-root/ls-hqvhost04/power"/>

 <configResolveDn dn="org-root/ls-hqvhost04/power" cookie="1437593293/31565a68-084f-4029-b2ba-e48a80e98419" response="yes"> <outConfig> <lsPower dn="org-root/ls-hqvhost04/power" state="up"/> </outConfig> </configResolveDn>

Status: ON
<aaaLogout inCookie="1437593293/31565a68-084f-4029-b2ba-e48a80e98419" />

 <aaaLogout cookie="" response="yes" outStatus="success"> </aaaLogout>

[root@rheltest ~]# fence_cisco_ucs --ip=10.0.10.22 --username=rhevm --password=mrhev --plug=host04 --ssl --ssl-secure -o status -vvv
Unable to connect/login to fencing device
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

Does the fence_cisco_ucs script not function with the secure option enabled and a https only UCS manager?

Comment 7 Robert McSwain 2015-08-24 15:01:44 UTC
I've asked the customer to try the ssl_insecure=1 option with their power management as recommended in https://bugzilla.redhat.com/1240873 and will report back with what they find.

Comment 8 Moran Goldboim 2015-09-16 08:48:02 UTC
(In reply to Robert McSwain from comment #7)
> I've asked the customer to try the ssl_insecure=1 option with their power
> management as recommended in https://bugzilla.redhat.com/1240873 and will
> report back with what they find.

Hi Robert, any update here?

Comment 11 Eli Mesika 2015-09-24 08:08:40 UTC

*** This bug has been marked as a duplicate of bug 1240873 ***

Comment 12 Red Hat Bugzilla 2023-09-14 03:02:44 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days