Bug 1163105

Summary: [RFE] Supporting search in User-Level API
Product: Red Hat Enterprise Virtualization Manager Reporter: sefi litmanovich <slitmano>
Component: ovirt-engineAssignee: Eli Mesika <emesika>
Status: CLOSED ERRATA QA Contact: Karolína Hajná <khajna>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.5.0CC: aburden, bazulay, emesika, iheim, juan.hernandez, khajna, lpeer, lsurette, nicolas, pstehlik, rbalakri, Rhev-m-bugs, sherold, slitmano, yeylon, ykaul
Target Milestone: ovirt-3.6.0-rcKeywords: FutureFeature
Target Release: 3.6.0Flags: sherold: Triaged+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Enhancement
Doc Text:
Search query filtering has been added to the REST API to match functionality of the Web Admin Portal.
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-03-09 20:51:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Infra RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1213937, 1132506    

Description sefi litmanovich 2014-11-12 12:06:17 UTC
Description of problem:

when trying to make a query for e.g. an event, trying to filter the result based on the test in description has inconsistent behaviour.

1. one issue is that it fails to find certain strings and other it does - I couldn't find any logic behind this inconsistency.

e.g. I want to look for an event where description is:
"Host {host_name} was started by Engine"

when I try:
curl -k -u admin@internal -i -H "Accept: application/xml" https://{engine_ip}/api/events\;from\=\?search\=Start

if finds events with Start or Starting in description.

when I use:

curl -k -u admin@internal -i -H "Accept: application/xml" https://{engine_ip}/api/events\;from\=\?search\=Started

or

curl -k -u admin@internal -i -H "Accept: application/xml" https://{engine_ip}/api/events\;from\=\?search\=Host

it doesn't find anything, just returns an empty list.

2. seconds issue is that the query also ignores percent encoding, if you want to search for a more complex sentence and use %20 for e.g. to encode space, result also fails to find anything, or finds results based only on first word.

but when you want to search one word and to filter based on event type, then it does work for e.g.:

curl -k -u admin@internal -i -H "Accept: application/xml" https://{engine_ip}/api/events\;from\=\?search\=Start%20type=605

So this inconsistency should be checked and fixed to enable more elaborate queries.


Version-Release number of selected component (if applicable):

rhevm-3.5.0-0.19.beta.el6ev.noarch

How reproducible:

always

Actual results:

should get a list filtered according to the query.


Expected results:

Provides the expected result randomly and without any apparent rule behind it. 

Additional info:

Comment 1 Juan Hernández 2014-11-12 14:19:17 UTC
The RESTAPI doesn't process the queries itself, it just sends them to the backend, so if something needs to be changed in query syntax or processing it should be in the backend.

Note also that the RESTAPI does respect URL encoding (%20 and similar), in fact this is done by the application server way before the value of the "search" parameter is given to the RESTAPI. So it is up to the backend to process the resulting characters (a space in this case). Looks like for the backend a space signals the end of a search term, not an space embedded in a search term. If you want to search for strings containing spaces you will probably need to use quotes, and encode them correctly. For example:

  ...?search=%22separated%20by%20spaces%22

Which will result in sending to the backend the string "separated by spaces" (including the quotes).

Comment 2 Oved Ourfali 2015-04-12 10:55:59 UTC
*** Bug 1155967 has been marked as a duplicate of this bug. ***

Comment 3 Karolína Hajná 2015-04-16 08:29:06 UTC
I couldn't find source of the "Host {host_name} was started by Engine" event to test this. How did you have to do to get it?

Comment 4 Juan Hernández 2015-04-16 13:44:07 UTC
*** Bug 1212483 has been marked as a duplicate of this bug. ***

Comment 5 Karolína Hajná 2015-04-21 13:29:24 UTC
Search still doesn't work properly.

I tried to search for "User" in "User admin@internal logged in." event. This doesn't work. However, when searching for "ser" only it works fine. 

Second case is when searching for name of one of my hosts - "host1" in "Installing Host host1. Stage: Setup validation." event. Just as in the first case, it finds nothing. Works fine while searching for "ost". 



Searching for string with space works correctly when quoted:

...?search=%22separated%20string%22

Comment 6 Juan Hernández 2015-04-21 15:14:13 UTC
The reason for the failure with "User" is that "User" has a special meaning when searching events, it means that you want the events of a particular user, and then you need to specify some matching criteria. For example:

  ?search=User%3dadmin

That should search for the events where some of the properties of the user is "admin". Unfortunately this seems to trigger a database error in the current master. It generates the following SQL query:

---8<---
SELECT * FROM (SELECT * FROM audit_log WHERE (audit_log_id IN (SELECT
distinct audit_log.audit_log_id FROM  audit_log   LEFT OUTER JOIN
vdc_users_with_tags ON audit_log.user_id=vdc_users_with_tags.user_id
WHERE  (  vdc_users_with_tags.department LIKE '%admin%' OR
vdc_users_with_tags.domain LIKE '%admin%' OR
vdc_users_with_tags.groups LIKE '%admin%' OR
vdc_users_with_tags.mla_role LIKE '%admin%' OR
vdc_users_with_tags.name LIKE '%admin%' OR  vdc_users_with_tags.role
LIKE '%admin%' OR  vdc_users_with_tags.surname LIKE '%admin%' OR
vdc_users_with_tags.tag_name LIKE '%admin%' OR
vdc_users_with_tags.username LIKE '%admin%' OR
vdc_users_with_tags.vm_pool_name LIKE '%admin%' )  AND not deleted))
ORDER BY audit_log_id DESC ) as T1 OFFSET (1 -1) LIMIT 2147483647
--->8---

And when executed that query generates the following error:

---8<---
statementcallback; bad sql grammar [select * from (select * from audit log where ( audit log id in (select distinct audit log.audit log id from  audit log   left outer join vdc users with tags on audit log.user id=vdc users with tags.user id    where  (  vdc users with tags.department like '%admin%' or  vdc users with tags.domain like '%admin%' or  vdc users with tags.groups like '%admin%' or  vdc users with tags.mla role like '%admin%' or  vdc users with tags.name like '%admin%' or  vdc users with tags.role like '%admin%' or  vdc users with tags.surname like '%admin%' or  vdc users with tags.tag name like '%admin%' or  vdc users with tags.username like '%admin%' or  vdc users with tags.vm pool name like '%admin%' )  and not deleted))  order by audit log id desc ) as t1 offset (1 -1) limit 2147483647]; nested exception is org.postgresql.util.psqlexception: error: column vdc users with tags.groups does not exist
position: 318
--->8---

Alternatively, instead of search inside all the properties of the user you can search in a specific one, for example the name:

  ?search=user.name%3dadmin

Finally, if what you want to do is search in the message of the event, then you can use the "message" property. For example, if you want to search the events that have the form "User whatever logged in" you can use the following search query:

  message="User * logged in."

That needs to be URL encoded, and it will look like this:

  ?search=message%3D%22User%20*%20logged%20in.%22

Comment 7 Oved Ourfali 2015-04-27 05:39:59 UTC
Based on last comment by Juan, can we move this to modified?

Comment 8 Karolína Hajná 2015-04-28 10:18:23 UTC
Yes, this can be moved to modified. 
Except the database error Juan posted everything work fine.

Comment 9 Oved Ourfali 2015-04-28 16:37:34 UTC
(In reply to Karolína Hajná from comment #8)
> Yes, this can be moved to modified. 
> Except the database error Juan posted everything work fine.

I meant verified. Not modified. My bad. Can you move it to verified?

Comment 10 Karolína Hajná 2015-04-29 06:36:59 UTC
On 3.6.0-0.0.master.20150427173543.git61dec8c.el6 searching for 
...events\;from\=\?search\=User%3dadmin
still causes following error:

HTTP/1.1 400 Bad Request
Date: Wed, 29 Apr 2015 06:27:42 GMT
Content-Type: application/xml
Content-Length: 1073
Vary: Accept-Encoding
Connection: close

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<fault>
    <reason>Operation Failed</reason>
    <detail>statementcallback; bad sql grammar [select * from (select * from audit log where ( audit log id in (select distinct audit log.audit log id from  audit log   left outer join vdc users with tags on audit log.user id=vdc users with tags.user id    where  (  vdc users with tags.department like '%admin%' or  vdc users with tags.domain like '%admin%' or  vdc users with tags.groups like '%admin%' or  vdc users with tags.mla role like '%admin%' or  vdc users with tags.name like '%admin%' or  vdc users with tags.role like '%admin%' or  vdc users with tags.surname like '%admin%' or  vdc users with tags.tag name like '%admin%' or  vdc users with tags.username like '%admin%' or  vdc users with tags.vm pool name like '%admin%' )  and not deleted))  order by audit log id desc ) as t1 offset (1 -1) limit 2147483647]; nested exception is org.postgresql.util.psqlexception: error: column vdc users with tags.groups does not exist
  position: 316</detail>
</fault>



Since the user "admin" exists and events related to this user as well, this is a bug. (In reply to Oved Ourfali from comment #9)

Comment 11 Eli Mesika 2015-05-13 10:00:19 UTC
IMO , bug reported in comment 10 should be opened separately and this RFE should be moved to verify

Comment 12 Eli Mesika 2015-05-17 11:00:14 UTC
Please verify this one and open a new BZ on the unrelated BZ reported in comment 10

Comment 13 Karolína Hajná 2015-05-25 13:14:42 UTC
The search doesn't work for user roles. I added a user to engine and set it's roles to UserRole and PowerUserRole (but assigned user role seem to have no effect on result).

The I used

curl -k -u {user@domain} -i -H "Accept: application/xml" https://{engine_ip}/api/vms\;from\=\?search\=*

which returned

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<fault>
    <reason>Operation Failed</reason>
    <detail>query execution failed due to insufficient permissions.</detail>
</fault>

For VMs this should return listed user's VMs. 

For any administrator role the search works correctly.

Comment 14 Eli Mesika 2015-05-26 11:18:17 UTC
The reason is that you are using a user query without setting the Filter to true in the header, that makes you get the permission error. 

You must set "Filter: true" in the header as follows 


curl -k -u {user@domain} -i -H "Accept: application/xml" -H "Filter: true" https://{engine_ip}/api/vms\;from\=\?search\=*

Moving this back to ON_QA

Comment 15 Karolína Hajná 2015-05-27 11:17:34 UTC
Verified on  3.6.0-0.0.master.20150519172222.git9a2e2b3.el7

Comment 17 errata-xmlrpc 2016-03-09 20:51:06 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-0376.html

Comment 18 Nicolas Ecarnot 2017-01-05 07:54:33 UTC
(In reply to Eli Mesika from comment #14)
> The reason is that you are using a user query without setting the Filter to
> true in the header, that makes you get the permission error. 
> 
> You must set "Filter: true" in the header as follows 
> 
> 
> curl -k -u {user@domain} -i -H "Accept: application/xml" -H "Filter: true"
> https://{engine_ip}/api/vms\;from\=\?search\=*
> 
> Moving this back to ON_QA

Hi Eli,

My colleague discovered that when using fence_rhevm as 'admin@internal', nothing is returned when querying the status of a VM.
Tweaking the code of fence_rhevm, forcing the filter header to 'true', allows to get the correct data.

AFAIK, the filter header was mandatory only for users : how comes it seems to matter also for admin?

Thank you.

Comment 19 Juan Hernández 2017-01-05 08:37:42 UTC
Nicolas, what version of ovirt-engine are you using exactly?

Comment 20 Juan Hernández 2017-01-05 09:22:49 UTC
I ask because if you are using version 4 of the engine then the default version of the API is also version 4, and the current fence_rhevm code doesn't work correctly with version 4 of the API, it needs to be fixed:

  Explicitly use version 3 of the oVirt API
  https://github.com/ClusterLabs/fence-agents/pull/71

Comment 21 Nicolas Ecarnot 2018-04-09 12:06:00 UTC
(In reply to Juan Hernández from comment #19)
> Nicolas, what version of ovirt-engine are you using exactly?

Hello,

I'm using fence-agents-rhevm-4.0.11-66.el7_4.3.x86_64 to query guests on oVirt 3.6.7.