Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1518315

Summary: GetCapabilitiesVDS fails after upgrade to 4.2 due to missing openvswitch
Product: [oVirt] vdsm Reporter: Evgheni Dereveanchin <ederevea>
Component: GeneralAssignee: Edward Haas <edwardh>
Status: CLOSED CURRENTRELEASE QA Contact: Michael Burman <mburman>
Severity: high Docs Contact:
Priority: high    
Version: 4.20.8CC: bugs, danken, ederevea, mburman
Target Milestone: ovirt-4.2.0Flags: rule-engine: ovirt-4.2+
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: v4.20.9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-12-20 10:51:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Network RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Evgheni Dereveanchin 2017-11-28 15:47:09 UTC
Description of problem:
 After enabling ovirt-4.2-pre repo on a ppc64le 4.1 CentOS host and starting an upgrade from the UI, the host fails to activate due to missing openvswitch binaries and stays in Initializing state

Version-Release number of selected component (if applicable):
vdsm-4.20.8-1.el7.centos.ppc64le

How reproducible:
reproduced on ppc64le so far

Steps to Reproduce:
1. enable 4.2-pre repo on an existing 4.1 host 
    yum install http://resources.ovirt.org/pub/yum-repo/ovirt-release42-pre.rpm
2. in the admin portal (4.1.7) select installation -> upgrade

Actual results:
 Host stuck in Initializing state with error:
  command GetCapabilitiesVDS failed: Internal JSON-RPC error: {'reason': "[Errno 2] No such file or directory: '/usr/share/openvswitch/scripts/ovs-ctl'"}

Expected results:
 Host upgraded successfully

Comment 1 Dan Kenigsberg 2017-11-28 15:52:58 UTC
Why is openvswitch missing, Evgeni? Do you know?

Comment 2 Evgheni Dereveanchin 2017-11-28 15:54:46 UTC
Hi Dan, I'm not sure but it's not installed on this ppc system and never was. Will attach the OTOPI log in a second.

Comment 4 Evgheni Dereveanchin 2017-11-28 16:08:50 UTC
Did some research: the system was delpoyed a year ago and was initially installed 
 from master (which is similar to 4.1-pre) and later upgraded to 4.1.6 using plain yum. Not back then not now did openvswitch get installed, nor is it a dependency for VDSM. In my opinion, if a VDSM does not depend on openvswitch, it should not fail when that is missing. Alternatively, it should be added as a dependency so that it's installed when upgrading from previous versions.

Comment 5 Evgheni Dereveanchin 2017-11-28 16:18:33 UTC
Checked another ppc64le hypervisor that we have and it's also lacking openvswitch while all x86_64 hosts seem to have it so this may be specific for this platform.

Snippet from vdsm.log for reference:

2017-11-28 16:10:39,450+0000 ERROR (jsonrpc/7) [jsonrpc.JsonRpcServer] Internal server error (__init__:611)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 606, in _handle_request
    res = method(**params)
  File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, in _dynamicMethod
    result = fn(*methodArgs)
  File "<string>", line 2, in getCapabilities
  File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method
    ret = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1317, in getCapabilities
    c = caps.get()
  File "/usr/lib/python2.7/site-packages/vdsm/host/caps.py", line 168, in get
    net_caps = supervdsm.getProxy().network_caps()
  File "/usr/lib/python2.7/site-packages/vdsm/supervdsm.py", line 53, in __call__
    return callMethod()
  File "/usr/lib/python2.7/site-packages/vdsm/supervdsm.py", line 51, in <lambda>
    **kwargs)
  File "<string>", line 2, in network_caps
  File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod
    raise convert_to_error(kind, result)
OSError: [Errno 2] No such file or directory: '/usr/share/openvswitch/scripts/ovs-ctl'


Installing openvswitch manually allows the host to immediately get activated and fixes the issue. Still, the upgrade (even using yum) should do that automatically.

Comment 6 Dan Kenigsberg 2017-11-28 20:25:53 UTC
We've fixed the immediate problem of Vdsm not responsing if openvswitch is not installed.

Would you file another bug for "require openvswitch on ppc, too"?

Comment 7 Michael Burman 2017-12-03 10:10:37 UTC
Does v4.20.9 equals vdsm-4.20.8-79.gitd2a37c7.el7.centos.x86_64 ??

Comment 8 Dan Kenigsberg 2017-12-03 10:30:58 UTC
$ git describe d2a37c7
v4.20.9

yes

Comment 9 Michael Burman 2017-12-03 10:49:40 UTC
Verified on - vdsm-4.20.8-79.gitd2a37c7.el7.centos.x86_64

Vdsm is responding if openvswitch is not installed.

Comment 10 Sandro Bonazzola 2017-12-20 10:51:52 UTC
This bugzilla is included in oVirt 4.2.0 release, published on Dec 20th 2017.

Since the problem described in this bug report should be
resolved in oVirt 4.2.0 release, published on Dec 20th 2017, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.