Bug 1810974 - ipmilan fencing fails with JSON-RPC error when password contains space
Summary: ipmilan fencing fails with JSON-RPC error when password contains space
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: vdsm
Classification: oVirt
Component: Core
Version: 4.30.40
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: ovirt-4.4.1
: 4.40.19
Assignee: Eli Mesika
QA Contact: Petr Matyáš
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-03-06 10:30 UTC by Netbulae
Modified: 2020-07-08 08:27 UTC (History)
4 users (show)

Fixed In Version: vdsm-4.40.19
Clone Of:
Environment:
Last Closed: 2020-07-08 08:27:05 UTC
oVirt Team: Infra
Embargoed:
mperina: ovirt-4.4?


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 109117 0 master MERGED fix JSON-RPC error when fencing fails 2020-06-19 10:09:32 UTC

Description Netbulae 2020-03-06 10:30:28 UTC
Description of problem:

We have ipmi's with a space in the password for the fence user, when testing the fencing I get:

[jsonrpc.JsonRpcServer] RPC call Host.fenceNode failed (error 1) in 0.12 seconds (__init__:312)

Removing the space from the password works fine.

Comment 1 Eli Mesika 2020-04-19 15:51:37 UTC
Actually I am getting this error for IPMI agent when passing a value with a SPACE in the middle for the user or the password.

For example , I passed "ad min" as the user name and it returned with the same error , The JSON request looks like that :

<JsonRpcRequest id: "c387a6e3-4ccb-438f-8eff-7b56be0f6fb0", method: Host.fenceNode, params: {addr=10.37.160.42, port=, agent=ipmilan, username=ad min   , password=*****, action=status}>

Error in vdsm.log of the proxy host used for the status operation :

 [jsonrpc.JsonRpcServer] RPC call Host.fenceNode failed (error 1) in 0.39 seconds (__init__:312)

Seems that the {<param>=<value>[[,]<param>=<value>]*} itself does not allow spaces in the <value> string

Comment 2 Eli Mesika 2020-04-20 00:25:48 UTC
This fails from REST API as well when the user or password of the fence agent includes a space 

POST --> http://laptop.emesika.com:8080/ovirt-engine/api/hosts/33cb1aa0-a631-4c29-944d-b32a724245e5/fence
with body of

<action>
  <fence_type>status</fence_type>
</action>

resulted in 

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<action>
  <fault>
    <reason>Internal JSON-RPC error</reason>
  </fault>
  <fence_type>status</fence_type>
  <status>failed</status>
</action>

Comment 3 Eli Mesika 2020-04-22 09:18:24 UTC
Seems that IPMI does not allow to access the agent with a password that includes a space 

I checked with both ipmi fence script frpm fence-agents package (fence_ipmi) and from ipmitool 
Both rejected the <status> operation when the user had a password that includes a space 

I see here 2 options 

1) investigate all supported agents for a password that includes a space and apply validation accordingly 
   This is complex and we currently have only IPMI agent for testing 

2) reject any password that includes a space in our host PM configuration with a friendly message 

How should we continue ?

Comment 4 Netbulae 2020-04-22 13:48:34 UTC
For us option 2 would be good enough.

I had been debugging the issue a bit as Supermicro allowed spaces and I couldn't find much about the JSON-RPC error.

A proper error message and documentation entry would have been sufficient.

Comment 5 Martin Perina 2020-04-22 14:23:46 UTC
(In reply to Eli Mesika from comment #3)
> Seems that IPMI does not allow to access the agent with a password that
> includes a space 
> 
> I checked with both ipmi fence script frpm fence-agents package (fence_ipmi)
> and from ipmitool 
> Both rejected the <status> operation when the user had a password that
> includes a space 
> 
> I see here 2 options 
> 
> 1) investigate all supported agents for a password that includes a space and
> apply validation accordingly 
>    This is complex and we currently have only IPMI agent for testing 

Yes, very time consuming and most probably dependent on specific hardware capabilities.

> 
> 2) reject any password that includes a space in our host PM configuration
> with a friendly message 

But unless we are verifying using above we can't be sure that spaces in password works fine for some fence agents. And also we might break existing setups which worked fine until this fix.

> 
> How should we continue ?

Why are we throwing 'Internal JSON-RPC error'? Shouldn't we fail some standard error like 'Fencing operation failed: <REASON>', where <REASON> is the actual error from fence_ipmilan:

$ fence_ipmilan -a ipmi-21 -l test1 -p 'test 1' -P -L operator -o status
2020-04-22 16:18:50,117 ERROR: Failed: Unable to obtain correct plug status or plug is not available

It doesn't tell you anything about space in password, but it's standard error message that we get form fencing-agent, so we should just show it to the user. And if it's not enough, we can just create RFE for fence-agents package to produce more meaningful errors ...

WDYT?

Comment 6 Eli Mesika 2020-04-27 07:32:08 UTC
(In reply to Martin Perina from comment #5)

> WDYT?

Yes, I think that we need to do 2 things here 

1) fix the code to return the error we get when using fence_ipmi or ipmitool directly with a password that includes spaces 

2) open a bug on IPMI , I don't think its a fence-agents issue since ipmitool also gives the "Unable to obtain correct plug status or plug is not available" message

Comment 7 Martin Perina 2020-04-27 09:02:55 UTC
(In reply to Eli Mesika from comment #6)
> (In reply to Martin Perina from comment #5)
> 
> > WDYT?
> 
> Yes, I think that we need to do 2 things here 
> 
> 1) fix the code to return the error we get when using fence_ipmi or ipmitool
> directly with a password that includes spaces 

Yes, we need to propagate the error output we get from VDSM if calling fence action failed:

https://github.com/oVirt/vdsm/blob/master/lib/vdsm/API.py#L1292

That could be our fix for 4.4

> 
> 2) open a bug on IPMI , I don't think its a fence-agents issue since
> ipmitool also gives the "Unable to obtain correct plug status or plug is not
> available" message

I'm not sure they are able to get better erro code, but this is an RFE, so we can expect results not earlier than RHEL 8.3. So this is a long run.

Comment 8 Petr Matyáš 2020-06-05 11:45:38 UTC
Verified on vdsm-4.40.19-1.el8ev.x86_64

Test of connection works even with space in password with ipmilan (not sure if there was some BZ for ipmi as it's not linked here).
More verbose logging is present on failure (even though ipmi doesn't provide useful information).

Comment 9 Sandro Bonazzola 2020-07-08 08:27:05 UTC
This bugzilla is included in oVirt 4.4.1 release, published on July 8th 2020.

Since the problem described in this bug report should be resolved in oVirt 4.4.1 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.