Bug 1262359 - broker from 1.2.6.1 fails with VDSM 4.17 cause getVolumePath is no more avaialble
broker from 1.2.6.1 fails with VDSM 4.17 cause getVolumePath is no more avaia...
Status: CLOSED CURRENTRELEASE
Product: ovirt-hosted-engine-ha
Classification: oVirt
Component: Broker (Show other bugs)
1.2.6.1
Unspecified Unspecified
unspecified Severity unspecified (vote)
: ovirt-3.5.5
: 1.2.7.2
Assigned To: Simone Tiraboschi
Nikolai Sednev
integration
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-09-11 09:35 EDT by Simone Tiraboschi
Modified: 2015-10-27 03:46 EDT (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-10-26 09:44:09 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
sbonazzo: ovirt‑3.5.z?
sbonazzo: blocker?
rule-engine: planning_ack?
sbonazzo: devel_ack+
rule-engine: testing_ack?


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 47141 None None None Never

  None (edit)
Description Simone Tiraboschi 2015-09-11 09:35:14 EDT
Description of problem:
broker from 1.2.6.1 from oVirt 3.5 fails with VDSM 4.17 from oVirt 3.6 cause getVolumePath is no more available in VDSM 4.17 

[root@c71het20150910 ~]# systemctl status ovirt-ha-broker -l
ovirt-ha-broker.service - oVirt Hosted Engine High Availability Communications Broker
   Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-broker.service; enabled)
   Active: active (running) since ven 2015-09-11 14:13:50 CEST; 2min 25s ago
  Process: 776 ExecStart=/usr/lib/systemd/systemd-ovirt-ha-broker start (code=exited, status=0/SUCCESS)
 Main PID: 1053 (ovirt-ha-broker)
   CGroup: /system.slice/ovirt-ha-broker.service
           └─1053 /usr/bin/python /usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker

set 11 14:15:23 c71het20150910.localdomain ovirt-ha-broker[1053]: ovirt-ha-broker ovirt_hosted_engine_ha.broker.listener.ConnectionHandler ERROR Error handling request, data: 'set-storage-domain VdsmBackend hosted-engine.lockspace=7B22696D6167655F75756964223A202266653830393138662D366437302D346134662D396430322D343561666539323861383434222C202270617468223A206E756C6C2C2022766F6C756D655F75756964223A202231623832353466342D336565642D343862612D623830392D383230366235343665333033227D sp_uuid=470797a0-0ffb-4a27-b0bc-ebdec929f075 dom_type=iscsi hosted-engine.metadata=7B22696D6167655F75756964223A202265333636333633622D653134382D343038382D626234652D373765666333303232386232222C202270617468223A206E756C6C2C2022766F6C756D655F75756964223A202239396438346333362D373161382D346266362D396531302D396434363033396566323637227D sd_uuid=5094102c-e7f9-4f34-9362-936a2887faf5'
                                                                  Traceback (most recent call last):
                                                                    File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 166, in handle
                                                                      data)
                                                                    File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 299, in _dispatch
                                                                      .set_storage_domain(client, sd_type, **options)
                                                                    File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 65, in set_storage_domain
                                                                      self._backends[client].connect()
                                                                    File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 376, in connect
                                                                      volume.volume_uuid
                                                                    File "/usr/lib64/python2.7/xmlrpclib.py", line 1224, in __call__
                                                                      return self.__send(self.__name, args)
                                                                    File "/usr/lib64/python2.7/xmlrpclib.py", line 1578, in __request
                                                                      verbose=self.__verbose
                                                                    File "/usr/lib64/python2.7/xmlrpclib.py", line 1264, in request
                                                                      return self.single_request(host, handler, request_body, verbose)
                                                                    File "/usr/lib64/python2.7/xmlrpclib.py", line 1297, in single_request
                                                                      return self.parse_response(response)
                                                                    File "/usr/lib/python2.7/site-packages/vdsm/vdscli.py", line 43, in wrapped_parse_response
                                                                      return old_parse_response(*args, **kwargs)
                                                                    File "/usr/lib64/python2.7/xmlrpclib.py", line 1473, in parse_response
                                                                      return u.close()
                                                                    File "/usr/lib64/python2.7/xmlrpclib.py", line 793, in close
                                                                      raise Fault(**self._stack[0])
                                                                  Fault: <Fault 1: '<type \'exceptions.Exception\'>:method "getVolumePath" is not supported'>



Version-Release number of selected component (if applicable):
ovirt-hosted-engine-ha.noarch          1.2.6.1-1.el7.centos           @ovirt-3.5
vdsm.noarch                            4.17.6-0.el7.centos            @ovirt-3.6

How reproducible:
100%

Steps to Reproduce:
1. deploy hosted-engine from 3.5
2. add oVirt 3.6 repo
3. upgrade vdsm* without upgrading ovirt-hosted-engine-ha

Actual results:
It let you update VDSM only but then it fails

Expected results:
You have to update both to be on the same API

Additional info:
Comment 1 Martin Sivák 2015-09-11 10:13:35 EDT
Hmm I am not sure what to do about this as we can't change already released packages.. we can add conflicts to 3.5.z or just document this.
Comment 2 Nir Soffer 2015-09-11 10:22:54 EDT
getVolumePath was removed about a year ago in this commit:

commit 5578d97c46cfbdc8019296d737655e5c274e738a
Author: Federico Simoncelli <fsimonce@redhat.com>
Date:   Tue Sep 23 13:04:39 2014 +0000

    hsm: remove obsolete getVolumePath verb
    
    We can remove the getVolumePath verb as it was never officially in use.
    
    Change-Id: I7d15dee9e805478f4b4d2bfcb7985484db77339e
    Signed-off-by: Federico Simoncelli <fsimonce@redhat.com>
    Reviewed-on: http://gerrit.ovirt.org/33353
    Reviewed-by: Nir Soffer <nsoffer@redhat.com>
    Reviewed-by: Dan Kenigsberg <danken@redhat.com>

This method was obsolete and was kept only for testing purposes;
prepareImage should be used instead.
Comment 3 Simone Tiraboschi 2015-09-11 10:23:13 EDT
Just introduce a new conflict for 3.5.5
Comment 4 Red Hat Bugzilla Rules Engine 2015-10-18 04:34:25 EDT
Bug tickets that are moved to testing must have target release set to make sure tester knows what to test. Please set the correct target release before moving to ON_QA.
Comment 5 Nikolai Sednev 2015-10-20 08:47:28 EDT
Hi Simone,
Please provide a bit more detailed info for reproduction:
1.I need to add 3.6 repos to the engine and upgrade it first, then to add 3.6 repos to the host and try to upgrade the vdsm only?
2.Or I have to leave the engine 3.5 and only upgrade the host's vdsm to 3.6 repos?
Comment 6 Nikolai Sednev 2015-10-20 08:59:30 EDT
If I'm trying to upgrade the engine first, then I'm failing with the:
Error: Package: rhevm-setup-plugin-ovirt-engine-3.6.0.1-0.1.el6.noarch (qa-latest)
           Requires: rhevm >= 3.6.0
           Installed: rhevm-3.5.5-0.1.el6ev.noarch (@rhev-3.5-el6)
               rhevm = 3.5.5-0.1.el6ev
Error: rhevm-setup-plugin-ovirt-engine conflicts with rhevm-3.5.5-0.1.el6ev.noarch
Comment 7 Nikolai Sednev 2015-10-20 09:18:05 EDT
My environment consisted from one host with RHEL7.1 and RHEVM3.5.5-0.1.el6.
For the host only, upgrading of the vdsm* worked for me with the following components:
libvirt-python-1.2.8-7.el7_1.1.x86_64
libvirt-daemon-driver-nodedev-1.2.8-16.el7_1.4.x86_64
libvirt-daemon-driver-storage-1.2.8-16.el7_1.4.x86_64
sanlock-python-3.2.2-2.el7.x86_64
libvirt-client-1.2.8-16.el7_1.4.x86_64
libvirt-daemon-config-nwfilter-1.2.8-16.el7_1.4.x86_64
libvirt-lock-sanlock-1.2.8-16.el7_1.4.x86_64
libvirt-daemon-driver-network-1.2.8-16.el7_1.4.x86_64
libvirt-daemon-kvm-1.2.8-16.el7_1.4.x86_64
ovirt-hosted-engine-ha-1.2.7.2-1.el7ev.noarch
vdsm-4.17.9-1.el7ev.noarch
sanlock-3.2.2-2.el7.x86_64
libvirt-daemon-1.2.8-16.el7_1.4.x86_64
libvirt-daemon-driver-secret-1.2.8-16.el7_1.4.x86_64
libvirt-daemon-driver-qemu-1.2.8-16.el7_1.4.x86_64
ovirt-hosted-engine-setup-1.2.6.1-1.el7ev.noarch
mom-0.5.1-1.el7ev.noarch
sanlock-lib-3.2.2-2.el7.x86_64
qemu-kvm-rhev-2.1.2-23.el7_1.10.x86_64
libvirt-daemon-driver-nwfilter-1.2.8-16.el7_1.4.x86_64
libvirt-daemon-driver-interface-1.2.8-16.el7_1.4.x86_64
Red Hat Enterprise Linux Server release 7.1 (Maipo)

Engine:
rhevm-3.5.5-0.1.el6ev.noarch
ovirt-host-deploy-1.3.2-1.el6ev.noarch
ovirt-host-deploy-java-1.3.2-1.el6ev.noarch
Red Hat Enterprise Linux Server release 6.7 (Santiago)


This is the outprint of ovirt-ha-broker after the upgrade of vdsm* on host:

systemctl status ovirt-ha-broker -l
ovirt-ha-broker.service - oVirt Hosted Engine High Availability Communications Broker
   Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-broker.service; enabled)
   Active: active (running) since Tue 2015-10-20 11:47:07 IDT; 1h 19min ago
 Main PID: 18130 (ovirt-ha-broker)
   CGroup: /system.slice/ovirt-ha-broker.service
           └─18130 /usr/bin/python /usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker

Oct 20 13:05:59 alma03.qa.lab.tlv.redhat.com ovirt-ha-broker[18130]: ovirt-ha-broker mem_free.MemFree ERROR Failed to getVdsStats: VDSM initialization timeout
Oct 20 13:06:06 alma03.qa.lab.tlv.redhat.com ovirt-ha-broker[18130]: ovirt-ha-broker mgmt_bridge.MgmtBridge ERROR Failed to getVdsCapabilities: VDSM initialization timeout
Oct 20 13:06:08 alma03.qa.lab.tlv.redhat.com ovirt-ha-broker[18130]: ovirt-ha-broker engine_health.CpuLoadNoEngine ERROR Failed to getVmStats: VDSM initialization timeout
Oct 20 13:06:09 alma03.qa.lab.tlv.redhat.com ovirt-ha-broker[18130]: ovirt-ha-broker mem_free.MemFree ERROR Failed to getVdsStats: VDSM initialization timeout
Oct 20 13:06:17 alma03.qa.lab.tlv.redhat.com ovirt-ha-broker[18130]: ovirt-ha-broker mgmt_bridge.MgmtBridge ERROR Failed to getVdsCapabilities: VDSM initialization timeout
Oct 20 13:06:18 alma03.qa.lab.tlv.redhat.com ovirt-ha-broker[18130]: ovirt-ha-broker engine_health.CpuLoadNoEngine ERROR Failed to getVmStats: VDSM initialization timeout
Oct 20 13:06:19 alma03.qa.lab.tlv.redhat.com ovirt-ha-broker[18130]: ovirt-ha-broker mem_free.MemFree ERROR Failed to getVdsStats: VDSM initialization timeout
Oct 20 13:06:27 alma03.qa.lab.tlv.redhat.com ovirt-ha-broker[18130]: ovirt-ha-broker mgmt_bridge.MgmtBridge ERROR Failed to getVdsCapabilities: VDSM initialization timeout
Oct 20 13:06:28 alma03.qa.lab.tlv.redhat.com ovirt-ha-broker[18130]: ovirt-ha-broker engine_health.CpuLoadNoEngine ERROR Failed to getVmStats: VDSM initialization timeout
Oct 20 13:06:29 alma03.qa.lab.tlv.redhat.com ovirt-ha-broker[18130]: ovirt-ha-broker mem_free.MemFree ERROR Failed to getVdsStats: VDSM initialization timeout


# systemctl restart ovirt-ha-broker && systemctl restart ovirt-ha-agent
# systemctl status ovirt-ha-broker -l
ovirt-ha-broker.service - oVirt Hosted Engine High Availability Communications Broker
   Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-broker.service; enabled)
   Active: active (running) since Tue 2015-10-20 13:07:22 IDT; 9s ago
  Process: 60543 ExecStop=/usr/lib/systemd/systemd-ovirt-ha-broker stop (code=exited, status=0/SUCCESS)
  Process: 60559 ExecStart=/usr/lib/systemd/systemd-ovirt-ha-broker start (code=exited, status=0/SUCCESS)
 Main PID: 60572 (ovirt-ha-broker)
   CGroup: /system.slice/ovirt-ha-broker.service
           └─60572 /usr/bin/python /usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker

Oct 20 13:07:22 alma03.qa.lab.tlv.redhat.com systemd-ovirt-ha-broker[60559]: Starting ovirt-ha-broker: [  OK  ]
Oct 20 13:07:22 alma03.qa.lab.tlv.redhat.com systemd[1]: Started oVirt Hosted Engine High Availability Communications Broker.
[root@alma03 ~]# systemctl status ovirt-ha-broker -l
ovirt-ha-broker.service - oVirt Hosted Engine High Availability Communications Broker
   Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-broker.service; enabled)
   Active: active (running) since Tue 2015-10-20 13:07:22 IDT; 24s ago
  Process: 60543 ExecStop=/usr/lib/systemd/systemd-ovirt-ha-broker stop (code=exited, status=0/SUCCESS)
  Process: 60559 ExecStart=/usr/lib/systemd/systemd-ovirt-ha-broker start (code=exited, status=0/SUCCESS)
 Main PID: 60572 (ovirt-ha-broker)
   CGroup: /system.slice/ovirt-ha-broker.service
           └─60572 /usr/bin/python /usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker

Oct 20 13:07:22 alma03.qa.lab.tlv.redhat.com systemd-ovirt-ha-broker[60559]: Starting ovirt-ha-broker: [  OK  ]
Oct 20 13:07:22 alma03.qa.lab.tlv.redhat.com systemd[1]: Started oVirt Hosted Engine High Availability Communications Broker.

During the upgrade of vdsm*, I've noticed that ovirt-hosted-engine-ha-1.2.7.2-1.el7ev.noarch remained the same, as it was already the latest before the upgrade and remained the same after it, so only vdsm* were upgraded and continued to work properly after restarting restart ovirt-ha-broker && restart ovirt-ha-agent on host.
Comment 8 Simone Tiraboschi 2015-10-20 09:39:21 EDT
(In reply to Nikolai Sednev from comment #5)
> Hi Simone,
> Please provide a bit more detailed info for reproduction:
> 1.I need to add 3.6 repos to the engine and upgrade it first, then to add
> 3.6 repos to the host and try to upgrade the vdsm only?
> 2.Or I have to leave the engine 3.5 and only upgrade the host's vdsm to 3.6
> repos?

It should be the same regarding this topic.
Comment 9 Sandro Bonazzola 2015-10-26 09:44:09 EDT
oVirt 3.5.5 has been released including fixes for this issue.

Note You need to log in before you can comment on or make changes to this bug.