Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1401421

Summary: [z-stream clone - 4.0.7] api.vms.get(name) sometimes returns None when the vm exists
Product: Red Hat Enterprise Virtualization Manager Reporter: rhev-integ
Component: ovirt-engineAssignee: Ravi Nori <rnori>
Status: CLOSED ERRATA QA Contact: Gonza <grafuls>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.6.6CC: bugs, gklein, lsurette, lsvaty, mbasti, mgoldboi, michal.skrivanek, mperina, oourfali, pspacek, pstehlik, pvoborni, rbalakri, Rhev-m-bugs, rnori, srevivo, ykaul
Target Milestone: ovirt-4.0.7Keywords: Reopened, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1301067 Environment:
Last Closed: 2017-03-16 15:30:06 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Infra RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1301067    
Bug Blocks:    

Description rhev-integ 2016-12-05 08:30:32 UTC
+++ This bug is a downstream clone. The original bug is: +++
+++   bug 1301067 +++
======================================================================

Description of problem:
"""
api = API(url=self.url,
          kerberos=True,
          ca_file=self.ca_file,
          filter=True
      )

name = "some-vm-name"
vm = api.vms.get(name)
vm2 = api.vms.get(name)
vm3 = api.vms.get(name)
"""

quite often(subjective, no real measurement) in can happen that one or more of vm, vm2 or vm3 contain None but others have the desired vm object.


Version-Release number of selected component (if applicable):
rpm -qa "*ovirt*"
ovirt-engine-sdk-java-javadoc-3.5.1.0-1.fc22.noarch
libgovirt-0.3.3-1.fc22.x86_64
ovirt-engine-sdk-python-3.6.0.3-1.fc22.noarch
ovirt-engine-cli-3.6.0.2-1.fc22.noarch

server is: Version 3.6.1-0.2.el6


How reproducible:
not always

Steps to Reproduce:
1. see description


Actual results:
api.vms.get(name) sometimes returns None when the vm exists

Expected results:
api.vms.get(name) always returns vm object if the vm exists

Additional info:
If it returns None because of some external error(network, auth issue). Then the API should raise an exception and not return None.

(Originally by Petr Vobornik)

Comment 1 rhev-integ 2016-12-05 08:30:44 UTC
same with:
  api.vms.list(query=query)

same call multiple times, different results

(Originally by Petr Vobornik)

Comment 2 rhev-integ 2016-12-05 08:30:52 UTC
Any chances you can repeat this with the "debug=True" parameter and share the results? Otherwise it is very difficult to determine if this is caused by a server side issue or by an SDK issue. Also, it would be helpful to have also the server side logs.

(Originally by juan.hernandez)

Comment 3 rhev-integ 2016-12-05 08:31:00 UTC
*** Bug 1301066 has been marked as a duplicate of this bug. ***

(Originally by juan.hernandez)

Comment 5 rhev-integ 2016-12-05 08:31:14 UTC
attachment 1117228 [details] contains third run of test(failed at 15th attempt). Ve can see there that the result was as follows - so probably issue on server:

 <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
 <vms/>

test:
        name = self.args.name
        for i in range(0, 30):
            vm = self.backend.api.vms.get(name)
            if not vm:
                print "ERROR: attempt failed: %s" % i
                return
            else:
                print "INFO: attempt successful: %s" % i
        return

- first run failed at 2nd attempt
- second run was successful - all 30 attempts

I don't have access to server logs.

(Originally by Petr Vobornik)

Comment 6 rhev-integ 2016-12-05 08:31:21 UTC
Yes, looks like a server side issue. Can you repeat the same test using basic (user name and password) authentication instead of Kerberos?

(Originally by juan.hernandez)

Comment 7 rhev-integ 2016-12-05 08:31:28 UTC
the API endpoint in the previous example can be used only with Kerberos auth. Luckily the same server has also different API url where basic auth is allowed.

There, I(user which were used with kerberos auth) don't have a the required rights, got:
  status: 400
  reason: Bad Request
  detail: query execution failed due to insufficient permissions.

But the test seems to pass with basic auth, when a different user which was given permissions for the vm is used.

I've tried 500 attempts with the different user and all were successful.

(Originally by Petr Vobornik)

Comment 8 rhev-integ 2016-12-05 08:31:36 UTC
OK, it is an server side authentication issue then, I'm moving to the AAA component.

(Originally by juan.hernandez)

Comment 9 rhev-integ 2016-12-05 08:31:43 UTC
Ravi - thoughts?

(Originally by Oved Ourfali)

Comment 10 rhev-integ 2016-12-05 08:31:50 UTC
Sounds like a network issue performing auth, I will try to reproduce it

(Originally by Ravi Nori)

Comment 11 rhev-integ 2016-12-05 08:31:57 UTC
This bug is marked for z-stream, yet the milestone is for a major version, therefore the milestone has been reset.
Please set the correct milestone or drop the z stream flag.

(Originally by rule-engine)

Comment 12 rhev-integ 2016-12-05 08:32:05 UTC
I am unable to reproduce this on Master and 3.6 branches (3.6.3 and 3.6.1).
From the attached test run output it does not look like negotiate is failing, just that empty vm list is returned.

Please check on the latest 3.6 and see if it is reproducible, also double check the users permissions.

(Originally by Ravi Nori)

Comment 14 rhev-integ 2016-12-05 08:32:19 UTC
The server was upgraded to 3.6.2.6-0.1.el6 in a meanwhile.

I can still reproduce it, see attachment 1128524 [details] 

Tried 3 times.
1. first 30 was OK.
2. failed on 12th attempt (attached)
3. failed on 52nd attempt (max attempt was raised)

I have no idea what user permissions to check. I'm not admin of the server.

(Originally by Petr Vobornik)

Comment 17 rhev-integ 2016-12-05 08:32:39 UTC
Do you use mod_auth_kerb? If yes, is it possible for you to test with mod_auth_gssapi?

(Originally by Ondra Machacek)

Comment 18 rhev-integ 2016-12-05 08:32:46 UTC
mod_auth_gssapi is available only on RHEL 7 but AFAIK RHEL 7 is not supported with RHEV-M so the answer is no, we cannot test it.

(Originally by Petr Spacek)

Comment 19 rhev-integ 2016-12-05 08:32:54 UTC
As it doesn't happen on any other environment, and we can't seem to reproduce it in any way, I'm closing this one as wontfix.
If you're able to reproduce in a new clean environment please let us know.

(Originally by Oved Ourfali)

Comment 26 rhev-integ 2016-12-05 08:33:41 UTC
Since we are unable to reproduce this on the listed environment I am closing this as WORKSFORME, please feel free to reopen if the problem reoccurs

(Originally by Ravi Nori)

Comment 27 rhev-integ 2016-12-05 08:33:49 UTC
FYI I tried the reproducer from comment 16(only had to change api uri and certificate because it changed since that time, it is updated on the vm) and it reproduced. 

So it si still present on Version 4.0.4.4-0.1.el7ev

If you have issue with it, just ping me.


Reproducer output:
./reproducer-1301067.py 
<sanitized part will all my vms>
Fetching pvoborni-f24-a-ipa1
INFO: attempt successful: 0
INFO: attempt successful: 1
INFO: attempt successful: 2
INFO: attempt successful: 3
INFO: attempt successful: 4
INFO: attempt successful: 5
INFO: attempt successful: 6
INFO: attempt successful: 7
INFO: attempt successful: 8
INFO: attempt successful: 9
INFO: attempt successful: 10
INFO: attempt successful: 11
INFO: attempt successful: 12
INFO: attempt successful: 13
INFO: attempt successful: 14
ERROR: attempt failed: 15

(Originally by Petr Vobornik)

Comment 28 rhev-integ 2016-12-05 08:33:56 UTC
This is most probably has the same cause as BZ1396833, so marking as dependency

(Originally by Martin Perina)

Comment 30 Gonza 2017-02-16 15:08:29 UTC
Verified with:
python-ovirt-engine-sdk4-4.1.0-1.el7ev.x86_64

Comment 32 errata-xmlrpc 2017-03-16 15:30:06 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2017-0542.html