Bug 1301067

Summary: api.vms.get(name) sometimes returns None when the vm exists
Product: Red Hat Enterprise Virtualization Manager Reporter: Petr Vobornik <pvoborni>
Component: ovirt-engineAssignee: Aleksei Slaikovskii <aslaikov>
Status: CLOSED ERRATA QA Contact: Aleksei Slaikovskii <aslaikov>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.6.6CC: bugs, gklein, lsurette, lsvaty, mbasti, mgoldboi, michal.skrivanek, mperina, oourfali, pspacek, pvoborni, rbalakri, Rhev-m-bugs, rnori, srevivo, ykaul
Target Milestone: ovirt-4.1.0-alphaKeywords: Reopened, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1401421 (view as bug list) Environment:
Last Closed: 2016-07-11 18:05:45 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Infra RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1396833    
Bug Blocks: 1213937, 1401421    
Attachments:
Description Flags
Verification log none

Description Petr Vobornik 2016-01-22 13:50:26 UTC
Description of problem:
"""
api = API(url=self.url,
          kerberos=True,
          ca_file=self.ca_file,
          filter=True
      )

name = "some-vm-name"
vm = api.vms.get(name)
vm2 = api.vms.get(name)
vm3 = api.vms.get(name)
"""

quite often(subjective, no real measurement) in can happen that one or more of vm, vm2 or vm3 contain None but others have the desired vm object.


Version-Release number of selected component (if applicable):
rpm -qa "*ovirt*"
ovirt-engine-sdk-java-javadoc-3.5.1.0-1.fc22.noarch
libgovirt-0.3.3-1.fc22.x86_64
ovirt-engine-sdk-python-3.6.0.3-1.fc22.noarch
ovirt-engine-cli-3.6.0.2-1.fc22.noarch

server is: Version 3.6.1-0.2.el6


How reproducible:
not always

Steps to Reproduce:
1. see description


Actual results:
api.vms.get(name) sometimes returns None when the vm exists

Expected results:
api.vms.get(name) always returns vm object if the vm exists

Additional info:
If it returns None because of some external error(network, auth issue). Then the API should raise an exception and not return None.

Comment 1 Petr Vobornik 2016-01-22 14:18:05 UTC
same with:
  api.vms.list(query=query)

same call multiple times, different results

Comment 2 Juan Hernández 2016-01-22 14:42:56 UTC
Any chances you can repeat this with the "debug=True" parameter and share the results? Otherwise it is very difficult to determine if this is caused by a server side issue or by an SDK issue. Also, it would be helpful to have also the server side logs.

Comment 3 Juan Hernández 2016-01-22 14:44:21 UTC
*** Bug 1301066 has been marked as a duplicate of this bug. ***

Comment 5 Petr Vobornik 2016-01-22 15:32:32 UTC
attachment 1117228 [details] contains third run of test(failed at 15th attempt). Ve can see there that the result was as follows - so probably issue on server:

 <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
 <vms/>

test:
        name = self.args.name
        for i in range(0, 30):
            vm = self.backend.api.vms.get(name)
            if not vm:
                print "ERROR: attempt failed: %s" % i
                return
            else:
                print "INFO: attempt successful: %s" % i
        return

- first run failed at 2nd attempt
- second run was successful - all 30 attempts

I don't have access to server logs.

Comment 6 Juan Hernández 2016-01-22 15:39:57 UTC
Yes, looks like a server side issue. Can you repeat the same test using basic (user name and password) authentication instead of Kerberos?

Comment 7 Petr Vobornik 2016-01-22 16:12:28 UTC
the API endpoint in the previous example can be used only with Kerberos auth. Luckily the same server has also different API url where basic auth is allowed.

There, I(user which were used with kerberos auth) don't have a the required rights, got:
  status: 400
  reason: Bad Request
  detail: query execution failed due to insufficient permissions.

But the test seems to pass with basic auth, when a different user which was given permissions for the vm is used.

I've tried 500 attempts with the different user and all were successful.

Comment 8 Juan Hernández 2016-01-25 11:22:40 UTC
OK, it is an server side authentication issue then, I'm moving to the AAA component.

Comment 9 Oved Ourfali 2016-01-25 11:53:15 UTC
Ravi - thoughts?

Comment 10 Ravi Nori 2016-01-25 15:59:53 UTC
Sounds like a network issue performing auth, I will try to reproduce it

Comment 11 Red Hat Bugzilla Rules Engine 2016-01-27 10:16:38 UTC
This bug is marked for z-stream, yet the milestone is for a major version, therefore the milestone has been reset.
Please set the correct milestone or drop the z stream flag.

Comment 12 Ravi Nori 2016-02-17 14:48:54 UTC
I am unable to reproduce this on Master and 3.6 branches (3.6.3 and 3.6.1).
From the attached test run output it does not look like negotiate is failing, just that empty vm list is returned.

Please check on the latest 3.6 and see if it is reproducible, also double check the users permissions.

Comment 14 Petr Vobornik 2016-02-19 12:02:10 UTC
The server was upgraded to 3.6.2.6-0.1.el6 in a meanwhile.

I can still reproduce it, see attachment 1128524 [details] 

Tried 3 times.
1. first 30 was OK.
2. failed on 12th attempt (attached)
3. failed on 52nd attempt (max attempt was raised)

I have no idea what user permissions to check. I'm not admin of the server.

Comment 17 Ondra Machacek 2016-03-14 11:05:28 UTC
Do you use mod_auth_kerb? If yes, is it possible for you to test with mod_auth_gssapi?

Comment 18 Petr Spacek 2016-03-21 08:38:16 UTC
mod_auth_gssapi is available only on RHEL 7 but AFAIK RHEL 7 is not supported with RHEV-M so the answer is no, we cannot test it.

Comment 19 Oved Ourfali 2016-04-07 13:06:12 UTC
As it doesn't happen on any other environment, and we can't seem to reproduce it in any way, I'm closing this one as wontfix.
If you're able to reproduce in a new clean environment please let us know.

Comment 26 Ravi Nori 2016-07-11 18:05:45 UTC
Since we are unable to reproduce this on the listed environment I am closing this as WORKSFORME, please feel free to reopen if the problem reoccurs

Comment 27 Petr Vobornik 2016-11-14 11:28:31 UTC
FYI I tried the reproducer from comment 16(only had to change api uri and certificate because it changed since that time, it is updated on the vm) and it reproduced. 

So it si still present on Version 4.0.4.4-0.1.el7ev

If you have issue with it, just ping me.


Reproducer output:
./reproducer-1301067.py 
<sanitized part will all my vms>
Fetching pvoborni-f24-a-ipa1
INFO: attempt successful: 0
INFO: attempt successful: 1
INFO: attempt successful: 2
INFO: attempt successful: 3
INFO: attempt successful: 4
INFO: attempt successful: 5
INFO: attempt successful: 6
INFO: attempt successful: 7
INFO: attempt successful: 8
INFO: attempt successful: 9
INFO: attempt successful: 10
INFO: attempt successful: 11
INFO: attempt successful: 12
INFO: attempt successful: 13
INFO: attempt successful: 14
ERROR: attempt failed: 15

Comment 28 Martin Perina 2016-12-02 14:08:30 UTC
This is most probably has the same cause as BZ1396833, so marking as dependency

Comment 31 Aleksei Slaikovskii 2017-02-01 11:44:22 UTC
Created attachment 1246617 [details]
Verification log

Verified on ovirt-engine-sdk-python-3.6.9.1