Bug 1450677

Summary: Fail to execute neutron functional metadata_proxy tests
Product: Red Hat OpenStack Reporter: Arie Bregman <abregman>
Component: openstack-neutronAssignee: Daniel Alvarez Sanchez <dalvarez>
Status: CLOSED ERRATA QA Contact: Toni Freger <tfreger>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 12.0 (Pike)CC: abregman, amuller, chrisw, dalvarez, ihrachys, mburns, nyechiel, sclewis, srevivo
Target Milestone: Upstream M2Keywords: AutomationBlocker, Triaged
Target Release: 12.0 (Pike)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-neutron-11.0.0-0.20170521040619.3f2e22a.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-12-13 21:27:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Arie Bregman 2017-05-14 14:53:19 UTC
Description of problem:

Neutron functional metadata_proxy tests fail:

Tests: 
neutron.tests.functional.agent.l3.test_metadata_proxy.UnprivilegedUserMetadataL3AgentTestCase.test_haproxy_migration_path

Failure:


Traceback (most recent call last):
  File "neutron/tests/base.py", line 115, in func
    return f(self, *args, **kwargs)
  File "neutron/tests/base.py", line 115, in func
    return f(self, *args, **kwargs)
  File "neutron/tests/functional/agent/l3/test_metadata_proxy.py", line 174, in test_haproxy_migration_path
    self.assertIn('haproxy', pm.cmdline)
  File "/usr/lib/python2.7/site-packages/testtools/testcase.py", line 356, in assertIn
    self.assertThat(haystack, Contains(needle), message)
  File "/usr/lib/python2.7/site-packages/testtools/testcase.py", line 435, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: 'haproxy' not in None

---
Test:

neutron.tests.functional.agent.l3.test_metadata_proxy.UnprivilegedUserMetadataL3AgentTestCase.test_access_to_metadata_proxy

Failure:

Traceback (most recent call last):
  File "neutron/tests/base.py", line 115, in func
    return f(self, *args, **kwargs)
  File "neutron/tests/base.py", line 115, in func
    return f(self, *args, **kwargs)
  File "neutron/tests/functional/agent/l3/test_metadata_proxy.py", line 120, in test_access_to_metadata_proxy
    firstline = self._query_metadata_proxy(machine)
  File "neutron/tests/functional/agent/l3/test_metadata_proxy.py", line 85, in _query_metadata_proxy
    self.fail('Timed out waiting metadata proxy to become available')
  File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 666, in fail
    raise self.failureException(msg)
AssertionError: Timed out waiting metadata proxy to become available
---

Version-Release number of selected component (if applicable): openstack-neutron-11.0.0-0.20170503012423.617fc60


How reproducible: 100%


Steps to Reproduce:
1. Install openstack-neutron
2. Run dsvm-functional tests

Actual results: The above failures


Expected results: Tests passed successfully

Comment 2 Red Hat Bugzilla Rules Engine 2017-05-14 14:56:58 UTC
This bugzilla has been removed from the release and needs to be reviewed and Triaged for another Target Release.

Comment 3 Assaf Muller 2017-05-15 10:36:04 UTC
Is 'haproxy' dependency not installed in functional testing? It is a dep in the spec file.

Comment 5 Ihar Hrachyshka 2017-05-18 16:35:43 UTC
In test log, we see:

InvalidUserOrGroupException: Invalid user/uid: '65534'

Apparently the nobody user is not there, or it does have a different number.

Comment 6 Daniel Alvarez Sanchez 2017-05-18 17:00:19 UTC
Current neutron functional tests assumes that there's a user with uid 65534 [0].
This, apparently, is always true upstream for the ubuntu xenial deployment in the gate. However, it's not in RHEL as the uuid 65534 is created by package nfs-utils which is not installed by default in our CI [1]:


$ rpm -q --scripts nfs-utils 
[...]
# Create nfsnobody uid as long as it does not already exist.
cat /etc/passwd | cut -d':' -f 1 | grep --quiet nfsnobody 2>/dev/null
if [ "$?" -eq 1 ]; then
    /usr/sbin/useradd -l -c "Anonymous NFS User" -r -g 65534 \
                -s /sbin/nologin -u 65534 -d /var/lib/nfs nfsnobody >/dev/null 2>&1 || :
else
   /usr/sbin/usermod -u 65534 -g 65534 nfsnobody >/dev/null 2>&1 || :


We should modify the functional tests so that it doesn't make that assumption.
I'll open a bug in launchpad and try to come up with a fix.

[0] https://github.com/openstack/neutron/blob/stable/ocata/neutron/tests/functional/agent/l3/test_metadata_proxy.py#L133
[1] https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/network/view/neutron/job/DFG-network-neutron-dsvm-functional-rhos/240/artifact/infrared/9795f228d997b171ef31983a4c7bf782-tester-0/home/cloud-user/logs/installed-rpms.log

Comment 7 Daniel Alvarez Sanchez 2017-05-19 07:58:57 UTC
Apparently, nfs-utils package gets installed as a dependency for libvirt:

$ repoquery --requires --recursive --resolve libvirt | grep nfs-utils
nfs-utils-1:1.3.0-0.33.el7_3.x86_64


Once nfs-utils is installed it creates the nfsnobody user with uid 65534 that allows functional tests to pass. In our CI we're not installing libvirt for functional tests [0] and since it's not a dependency either I guess we shouldn't relay on this uid to exist on the system for the tests.

[0] https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/network/view/neutron/job/DFG-network-neutron-dsvm-functional-rhos/240/artifact/infrared/9795f228d997b171ef31983a4c7bf782-tester-0/home/cloud-user/logs/installed-rpms.log

Comment 8 Daniel Alvarez Sanchez 2017-05-23 13:35:05 UTC
nfs-utils has been added as a requirement to python-neutron-tests [0] and it's already in RDO promotion that happened today. It should be auto-imported to OSP in a few days. I'll update the fields accordingly.

[0] https://review.rdoproject.org/r/#/c/6708/

Comment 9 Ihar Hrachyshka 2017-05-26 18:19:10 UTC
The fix landed in RDO, we are waiting for OSP sync to happen to close the bug.

Comment 15 Assaf Muller 2017-07-14 19:52:52 UTC
Verified per comment 12.

Comment 18 errata-xmlrpc 2017-12-13 21:27:05 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462