Description of problem: Unable to upgrade to RHVH rhvh-4.2.3.0-0.20180508.0 Version-Release number of selected component (if applicable): RHV 4.2 RHV H- rhvh-4.2.3.0-0.20180508.0 How reproducible: Always Steps to Reproduce: 1. Upgraded Manager to 4.2 successfully 2. Try to upgrade host from the manager. 3. It failed with error: unable to upgrade the host. 4. Also tried manually from command to update the package redhat-virtualization-host-image-update. error: # yum install redhat-virtualization-host-image-update-4.2-20180508.0.el7_5.noarch -y Loaded plugins: imgbased-persist, package_upload, product-id, search-disabled-repos, subscription-manager Resolving Dependencies --> Running transaction check ---> Package redhat-virtualization-host-image-update.noarch 0:4.2-20180508.0.el7_5 will be installed --> Finished Dependency Resolution Dependencies Resolved =================================================================================================================================================================== Package Arch Version Repository Size =================================================================================================================================================================== Installing: redhat-virtualization-host-image-update noarch 4.2-20180508.0.el7_5 rhel-7-server-rhvh-4-rpms 613 M Transaction Summary ==================================================================================================================================================================== Install 1 Package Total download size: 613 M Installed size: 612 M Downloading packages: redhat-virtualization-host-image-update-4.2-20180508.0.el7_5.noarch.rpm | 613 MB 00:00:35 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : redhat-virtualization-host-image-update-4.2-20180508.0.el7_5.noarch 1/1 warning: %post(redhat-virtualization-host-image-update-4.2-20180508.0.el7_5.noarch) scriptlet failed, exit status 1 Non-fatal POSTIN scriptlet failure in rpm package redhat-virtualization-host-image-update-4.2-20180508.0.el7_5.noarch Uploading Package Profile Verifying : redhat-virtualization-host-image-update-4.2-20180508.0.el7_5.noarch 1/1 Installed: redhat-virtualization-host-image-update.noarch 0:4.2-20180508.0.el7_5 Complete! ============================================= The layer is showing but can't boot as not in menu entry: # nodectl info layers: rhvh-4.1-0.20180126.0: rhvh-4.1-0.20180126.0+1 rhvh-4.2.3.0-0.20180508.0: rhvh-4.1-0.20180425.0: rhvh-4.1-0.20180425.0+1 bootloader: default: rhvh-4.1-0.20180425.0+1 entries: rhvh-4.1-0.20180126.0+1: index: 1 title: rhvh-4.1-0.20180126.0 kernel: /boot/rhvh-4.1-0.20180126.0+1/vmlinuz-3.10.0-693.17.1.el7.x86_64 args: "ro crashkernel=auto rd.lvm.lv=rhvh/rhvh-4.1-0.20180126.0+1 rd.lvm.lv=rhvh/swap rhgb quiet LANG=en_US.UTF-8 img.bootid=rhvh-4.1-0.20180126.0+1" initrd: /boot/rhvh-4.1-0.20180126.0+1/initramfs-3.10.0-693.17.1.el7.x86_64.img root: /dev/rhvh/rhvh-4.1-0.20180126.0+1 rhvh-4.1-0.20180425.0+1: index: 0 title: rhvh-4.1-0.20180425.0 kernel: /boot/rhvh-4.1-0.20180425.0+1/vmlinuz-3.10.0-862.el7.x86_64 args: "ro crashkernel=auto rd.lvm.lv=rhvh/swap rd.lvm.lv=rhvh/rhvh-4.1-0.20180425.0+1 rhgb quiet LANG=en_US.UTF-8 img.bootid=rhvh-4.1-0.20180425.0+1" initrd: /boot/rhvh-4.1-0.20180425.0+1/initramfs-3.10.0-862.el7.x86_64.img root: /dev/rhvh/rhvh-4.1-0.20180425.0+1 current_layer: rhvh-4.1-0.20180425.0+1 ================================================ /var/log/messages: May 17 09:42:53 dhcp210-242 python: ansible-yum Invoked with allow_downgrade=False name=['ovirt-host'] validate_certs=True list=None disable_gpg_check=False conf_file=None install_repoquery=True state=latest disablerepo=None update_cache=True disable_plugin=[] enablerepo=None exclude=None security=False update_only=False enable_plugin=[] installroot=/ skip_broken=False May 17 09:43:07 dhcp210-242 python: ansible-yum Invoked with allow_downgrade=False name=['*'] validate_certs=True list=None disable_gpg_check=False conf_file=None install_repoquery=True state=latest disablerepo=None update_cache=True disable_plugin=[] enablerepo=None exclude=None security=False update_only=False enable_plugin=[] installroot=/ skip_broken=False May 17 09:44:15 dhcp210-242 systemd-logind: Removed session 4. May 17 09:44:36 dhcp210-242 kernel: loop: module loaded May 17 09:44:37 dhcp210-242 kernel: squashfs: version 4.0 (2009/01/31) Phillip Lougher May 17 09:44:37 dhcp210-242 kernel: EXT4-fs (loop1): mounted filesystem with ordered data mode. Opts: (null) May 17 09:44:37 dhcp210-242 imgbased: 2018-05-17 09:44:37,320 [INFO] (MainThread) Extracting image '/usr/share/redhat-virtualization-host/image//redhat-virtualization-host-4.2-20180508.0.el7_5.squashfs.img' May 17 09:44:37 dhcp210-242 kernel: EXT4-fs (loop3): mounted filesystem with ordered data mode. Opts: (null) May 17 09:44:37 dhcp210-242 imgbased: 2018-05-17 09:44:37,497 [INFO] (MainThread) Starting base creation May 17 09:44:37 dhcp210-242 imgbased: 2018-05-17 09:44:37,497 [INFO] (MainThread) New base will be: rhvh-4.2.3.0-0.20180508.0 May 17 09:44:37 dhcp210-242 python: detected unhandled Python exception in '/tmp/tmp.wib3Mcwa4l/usr/lib/python2.7/site-packages/imgbased/__main__.py' May 17 09:44:38 dhcp210-242 abrt-server: Executable '/tmp/tmp.wib3Mcwa4l/usr/lib/python2.7/site-packages/imgbased/__main__.py' doesn't belong to any package and ProcessUnpackaged is set to 'no' May 17 09:44:38 dhcp210-242 abrt-server: 'post-create' on '/var/tmp/abrt/Python-2018-05-17-09:44:37-22229' exited with 1 May 17 09:44:38 dhcp210-242 abrt-server: Deleting problem directory '/var/tmp/abrt/Python-2018-05-17-09:44:37-22229' May 17 09:44:38 dhcp210-242 yum[21988]: Installed: redhat-virtualization-host-image-update-4.2-20180508.0.el7_5.noarch =============================================== Actual results: Failed to upgrade to 4.2 Expected results: Successfully upgrade the host. Additional info:
Tried with another node too: Failed with same error: node 2 May 17 10:16:28 dhcp209-217 python: ansible-yum Invoked with allow_downgrade=False name=['ovirt-host'] validate_certs=True list=None disable_gpg_check=False conf_file=None install_repoquery=True state=latest disablerepo=None update_cache=True disable_plugin=[] enablerepo=None exclude=None security=False update_only=False enable_plugin=[] installroot=/ skip_broken=False May 17 10:16:50 dhcp209-217 python: ansible-yum Invoked with allow_downgrade=False name=['*'] validate_certs=True list=None disable_gpg_check=False conf_file=None install_repoquery=True state=latest disablerepo=None update_cache=True disable_plugin=[] enablerepo=None exclude=None security=False update_only=False enable_plugin=[] installroot=/ skip_broken=False May 17 10:17:31 dhcp209-217 systemd: Started Session 510 of user root. May 17 10:17:31 dhcp209-217 systemd-logind: New session 510 of user root. May 17 10:17:31 dhcp209-217 systemd: Starting Session 510 of user root. May 17 10:19:02 dhcp209-217 kernel: loop: module loaded May 17 10:19:02 dhcp209-217 kernel: squashfs: version 4.0 (2009/01/31) Phillip Lougher May 17 10:19:02 dhcp209-217 kernel: EXT4-fs (loop1): mounted filesystem with ordered data mode. Opts: (null) May 17 10:19:02 dhcp209-217 imgbased: 2018-05-17 10:19:02,600 [INFO] (MainThread) Extracting image '/usr/share/redhat-virtualization-host/image//redhat-virtualization-host-4.2-20180508.0.el7_5.squashfs.img' May 17 10:19:02 dhcp209-217 kernel: EXT4-fs (loop3): mounted filesystem with ordered data mode. Opts: (null) May 17 10:19:03 dhcp209-217 imgbased: 2018-05-17 10:19:03,128 [INFO] (MainThread) Starting base creation May 17 10:19:03 dhcp209-217 imgbased: 2018-05-17 10:19:03,128 [INFO] (MainThread) New base will be: rhvh-4.2.3.0-0.20180508.0 May 17 10:19:04 dhcp209-217 imgbased: 2018-05-17 10:19:04,059 [INFO] (MainThread) New LV is: <LV 'rhvh/rhvh-4.2.3.0-0.20180508.0' /> May 17 10:19:04 dhcp209-217 dmeventd[824]: No longer monitoring thin pool rhvh-pool00-tpool. May 17 10:19:04 dhcp209-217 lvm[824]: Monitoring thin pool rhvh-pool00-tpool. May 17 10:19:04 dhcp209-217 multipathd: dm-13: remove map (uevent) May 17 10:19:04 dhcp209-217 multipathd: dm-13: devmap not registered, can't remove May 17 10:19:04 dhcp209-217 multipathd: dm-13: remove map (uevent) May 17 10:19:05 dhcp209-217 imgbased: 2018-05-17 10:19:05,420 [INFO] (MainThread) Creating new filesystem on base May 17 10:19:08 dhcp209-217 imgbased: 2018-05-17 10:19:08,437 [INFO] (MainThread) Writing tree to base May 17 10:19:08 dhcp209-217 kernel: EXT4-fs (dm-13): mounted filesystem with ordered data mode. Opts: (null) May 17 10:20:02 dhcp209-217 systemd: Started Session 511 of user root. May 17 10:20:02 dhcp209-217 systemd: Starting Session 511 of user root. May 17 10:21:43 dhcp209-217 kernel: SELinux: Context system_u:object_r:ovirt_vmconsole_exec_t:s0 is not valid (left unmapped). May 17 10:21:54 dhcp209-217 dmeventd[824]: No longer monitoring thin pool rhvh-pool00-tpool. May 17 10:21:54 dhcp209-217 lvm[824]: Monitoring thin pool rhvh-pool00-tpool. May 17 10:21:55 dhcp209-217 multipathd: dm-13: remove map (uevent) May 17 10:21:55 dhcp209-217 multipathd: dm-13: devmap not registered, can't remove May 17 10:21:55 dhcp209-217 multipathd: dm-13: remove map (uevent) May 17 10:21:55 dhcp209-217 imgbased: 2018-05-17 10:21:55,428 [INFO] (MainThread) Adding a new layer after <Base rhvh-4.2.3.0-0.20180508.0 [] /> May 17 10:21:55 dhcp209-217 imgbased: 2018-05-17 10:21:55,428 [INFO] (MainThread) Adding a new layer after <Base rhvh-4.2.3.0-0.20180508.0 [] /> May 17 10:21:55 dhcp209-217 imgbased: 2018-05-17 10:21:55,429 [INFO] (MainThread) New layer will be: <Layer rhvh-4.2.3.0-0.20180508.0+1 /> May 17 10:21:55 dhcp209-217 dmeventd[824]: No longer monitoring thin pool rhvh-pool00-tpool. May 17 10:21:55 dhcp209-217 lvm[824]: Monitoring thin pool rhvh-pool00-tpool. May 17 10:21:58 dhcp209-217 kernel: EXT4-fs (dm-14): mounted filesystem with ordered data mode. Opts: (null) May 17 10:21:58 dhcp209-217 kernel: EXT4-fs (dm-14): mounted filesystem with ordered data mode. Opts: (null) May 17 10:21:59 dhcp209-217 kernel: EXT4-fs (dm-14): mounted filesystem with ordered data mode. Opts: (null) May 17 10:22:00 dhcp209-217 kernel: EXT4-fs (dm-12): mounted filesystem with ordered data mode. Opts: (null) May 17 10:22:05 dhcp209-217 dbus[1006]: [system] Reloaded configuration May 17 10:22:13 dhcp209-217 imgbased: 2018-05-17 10:22:13,213 [INFO] (remediate_etc) Syncing systemd levels May 17 10:22:13 dhcp209-217 dbus[1006]: [system] Reloaded configuration May 17 10:22:14 dhcp209-217 kernel: EXT4-fs (dm-14): mounted filesystem with ordered data mode. Opts: (null) May 17 10:22:21 dhcp209-217 imgbased: 2018-05-17 10:22:21,571 [INFO] (remediate_etc) Syncing systemd levels May 17 10:22:22 dhcp209-217 kernel: EXT4-fs (dm-14): mounted filesystem with ordered data mode. Opts: (null) May 17 10:22:22 dhcp209-217 kernel: device-mapper: thin: Data device (dm-2) discard unsupported: Disabling discard passdown. May 17 10:22:26 dhcp209-217 kernel: EXT4-fs (dm-15): mounted filesystem with ordered data mode. Opts: (null) May 17 10:22:29 dhcp209-217 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.submonitor_base.SubmonitorBase ERROR Error executing submonitor engine-health, args {'use_ssl': 'true', 'vm_uuid': '5a1dab2d-c19f-4e51-a225-71882c535084', 'address': '0'}#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitor_base.py", line 115, in _worker#012 self.action(self._options)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitors/engine_health.py", line 49, in action#012 logger=self._log#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/util.py", line 187, in connect_vdsm_json_rpc#012 requestQueue=requestQueue,#012 File "/usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py", line 248, in connect#012 responseQueue)#012 File "/usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py", line 234, in _create#012 lazy_start=False)#012 File "/usr/lib/python2.7/site-packages/yajsonrpc/stompreactor.py", line 629, in StandAloneRpcClient#012 client = StompClient(utils.create_connected_socket(host, port, sslctx),#012 File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 947, in create_connected_socket#012 sock = sslctx.wrapSocket(sock)#012 File "/usr/lib/python2.7/site-packages/vdsm/sslutils.py", line 101, in wrapSocket#012 ca_certs=self.ca_certs)#012 File "/usr/lib64/python2.7/ssl.py", line 936, in wrap_socket#012 ciphers=ciphers)#012 File "/usr/lib64/python2.7/ssl.py", line 547, in __init__#012 self._context.load_verify_locations(ca_certs)#012IOError: [Errno 2] No such file or directory May 17 10:22:29 dhcp209-217 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.submonitor_base.SubmonitorBase ERROR Error executing submonitor mem-free, args {'use_ssl': 'true', 'address': '0'}#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitor_base.py", line 115, in _worker#012 self.action(self._options)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitors/mem_free.py", line 38, in action#012 logger=self._log#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/util.py", line 187, in connect_vdsm_json_rpc#012 requestQueue=requestQueue,#012 File "/usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py", line 248, in connect#012 responseQueue)#012 File "/usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py", line 234, in _create#012 lazy_start=False)#012 File "/usr/lib/python2.7/site-packages/yajsonrpc/stompreactor.py", line 629, in StandAloneRpcClient#012 client = StompClient(utils.create_connected_socket(host, port, sslctx),#012 File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 947, in create_connected_socket#012 sock = sslctx.wrapSocket(sock)#012 File "/usr/lib/python2.7/site-packages/vdsm/sslutils.py", line 101, in wrapSocket#012 ca_certs=self.ca_certs)#012 File "/usr/lib64/python2.7/ssl.py", line 936, in wrap_socket#012 ciphers=ciphers)#012 File "/usr/lib64/python2.7/ssl.py", line 547, in __init__#012 self._context.load_verify_locations(ca_certs)#012IOError: [Errno 2] No such file or directory May 17 10:22:30 dhcp209-217 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.submonitor_base.SubmonitorBase ERROR Error executing submonitor mgmt-bridge, args {'use_ssl': 'true', 'bridge_name': 'ovirtmgmt', 'address': '0'}#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitor_base.py", line 115, in _worker#012 self.action(self._options)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitors/mgmt_bridge.py", line 42, in action#012 logger=self._log#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/util.py", line 187, in connect_vdsm_json_rpc#012 requestQueue=requestQueue,#012 File "/usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py", line 248, in connect#012 responseQueue)#012 File "/usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py", line 234, in _create#012 lazy_start=False)#012 File "/usr/lib/python2.7/site-packages/yajsonrpc/stompreactor.py", line 629, in StandAloneRpcClient#012 client = StompClient(utils.create_connected_socket(host, port, sslctx),#012 File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 947, in create_connected_socket#012 sock = sslctx.wrapSocket(sock)#012 File "/usr/lib/python2.7/site-packages/vdsm/sslutils.py", line 101, in wrapSocket#012 ca_certs=self.ca_certs)#012 File "/usr/lib64/python2.7/ssl.py", line 936, in wrap_socket#012 ciphers=ciphers)#012 File "/usr/lib64/python2.7/ssl.py", line 547, in __init__#012 self._context.load_verify_locations(ca_certs)#012IOError: [Errno 2] No such file or directory May 17 10:22:38 dhcp209-217 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.submonitor_base.SubmonitorBase ERROR Error executing submonitor engine-health, args {'use_ssl': 'true', 'vm_uuid': '5a1dab2d-c19f-4e51-a225-71882c535084', 'address': '0'}#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitor_base.py", line 115, in _worker#012 self.action(self._options)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitors/engine_health.py", line 49, in action#012 logger=self._log#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/util.py", line 187, in connect_vdsm_json_rpc#012 requestQueue=requestQueue,#012 File "/usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py", line 248, in connect#012 responseQueue)#012 File "/usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py", line 234, in _create#012 lazy_start=False)#012 File "/usr/lib/python2.7/site-packages/yajsonrpc/stompreactor.py", line 629, in StandAloneRpcClient#012 client = StompClient(utils.create_connected_socket(host, port, sslctx),#012 File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 947, in create_connected_socket#012 sock = sslctx.wrapSocket(sock)#012 File "/usr/lib/python2.7/site-packages/vdsm/sslutils.py", line 101, in wrapSocket#012 ca_certs=self.ca_certs)#012 File "/usr/lib64/python2.7/ssl.py", line 936, in wrap_socket#012 ciphers=ciphers)#012 File "/usr/lib64/python2.7/ssl.py", line 547, in __init__#012 self._context.load_verify_locations(ca_certs)#012IOError: [Errno 2] No such file or directory May 17 10:22:38 dhcp209-217 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.submonitor_base.SubmonitorBase ERROR Error executing submonitor mem-free, args {'use_ssl': 'true', 'address': '0'}#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitor_base.py", line 115, in _worker#012 self.action(self._options)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitors/mem_free.py", line 38, in action#012 logger=self._log#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/util.py", line 187, in connect_vdsm_json_rpc#012 requestQueue=requestQueue,#012 File "/usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py", line 248, in connect#012 responseQueue)#012 File "/usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py", line 234, in _create#012 lazy_start=False)#012 File "/usr/lib/python2.7/site-packages/yajsonrpc/stompreactor.py", line 629, in StandAloneRpcClient#012 client = StompClient(utils.create_connected_socket(host, port, sslctx),#012 File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 947, in create_connected_socket#012 sock = sslctx.wrapSocket(sock)#012 File "/usr/lib/python2.7/site-packages/vdsm/sslutils.py", line 101, in wrapSocket#012 ca_certs=self.ca_certs)#012 File "/usr/lib64/python2.7/ssl.py", line 936, in wrap_socket#012 ciphers=ciphers)#012 File "/usr/lib64/python2.7/ssl.py", line 547, in __init__#012 self._context.load_verify_locations(ca_certs)#012IOError: [Errno 2] No such file or directory May 17 10:22:40 dhcp209-217 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.submonitor_base.SubmonitorBase ERROR Error executing submonitor mgmt-bridge, args {'use_ssl': 'true', 'bridge_name': 'ovirtmgmt', 'address': '0'}#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitor_base.py", line 115, in _worker#012 self.action(self._options)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitors/mgmt_bridge.py", line 42, in action#012 logger=self._log#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/util.py", line 187, in connect_vdsm_json_rpc#012 requestQueue=requestQueue,#012 File "/usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py", line 248, in connect#012 responseQueue)#012 File "/usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py", line 234, in _create#012 lazy_start=False)#012 File "/usr/lib/python2.7/site-packages/yajsonrpc/stompreactor.py", line 629, in StandAloneRpcClient#012 client = StompClient(utils.create_connected_socket(host, port, sslctx),#012 File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 947, in create_connected_socket#012 sock = sslctx.wrapSocket(sock)#012 File "/usr/lib/python2.7/site-packages/vdsm/sslutils.py", line 101, in wrapSocket#012 ca_certs=self.ca_certs)#012 File "/usr/lib64/python2.7/ssl.py", line 936, in wrap_socket#012 ciphers=ciphers)#012 File "/usr/lib64/python2.7/ssl.py", line 547, in __init__#012 self._context.load_verify_locations(ca_certs)#012IOError: [Errno 2] No such file or directory May 17 10:22:48 dhcp209-217 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.submonitor_base.SubmonitorBase ERROR Error executing submonitor engine-health, args {'use_ssl': 'true', 'vm_uuid': '5a1dab2d-c19f-4e51-a225-71882c535084', 'address': '0'}#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitor_base.py", line 115, in _worker#012 self.action(self._options)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitors/engine_health.py", line 49, in action#012 logger=self._log#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/util.py", line 187, in connect_vdsm_json_rpc#012 requestQueue=requestQueue,#012 File "/usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py", line 248, in connect#012 responseQueue)#012 File "/usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py", line 234, in _create#012 lazy_start=False)#012 File "/usr/lib/python2.7/site-packages/yajsonrpc/stompreactor.py", line 629, in StandAloneRpcClient#012 client = StompClient(utils.create_connected_socket(host, port, sslctx),#012 File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 947, in create_connected_socket#012 sock = sslctx.wrapSocket(sock)#012 File "/usr/lib/python2.7/site-packages/vdsm/sslutils.py", line 101, in wrapSocket#012 ca_certs=self.ca_certs)#012 File "/usr/lib64/python2.7/ssl.py", line 936, in wrap_socket#012 ciphers=ciphers)#012 File "/usr/lib64/python2.7/ssl.py", line 547, in __init__#012 self._context.load_verify_locations(ca_certs)#012IOError: [Errno 2] No such file or directory May 17 10:22:48 dhcp209-217 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.submonitor_base.SubmonitorBase ERROR Error executing submonitor mem-free, args {'use_ssl': 'true', 'address': '0'}#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitor_base.py", line 115, in _worker#012 self.action(self._options)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitors/mem_free.py", line 38, in action#012 logger=self._log#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/util.py", line 187, in connect_vdsm_json_rpc#012 requestQueue=requestQueue,#012 File "/usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py", line 248, in connect#012 responseQueue)#012 File "/usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py", line 234, in _create#012 lazy_start=False)#012 File "/usr/lib/python2.7/site-packages/yajsonrpc/stompreactor.py", line 629, in StandAloneRpcClient#012 client = StompClient(utils.create_connected_socket(host, port, sslctx),#012 File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 947, in create_connected_socket#012 sock = sslctx.wrapSocket(sock)#012 File "/usr/lib/python2.7/site-packages/vdsm/sslutils.py", line 101, in wrapSocket#012 ca_certs=self.ca_certs)#012 File "/usr/lib64/python2.7/ssl.py", line 936, in wrap_socket#012 ciphers=ciphers)#012 File "/usr/lib64/python2.7/ssl.py", line 547, in __init__#012 self._context.load_verify_locations(ca_certs)#012IOError: [Errno 2] No such file or directory May 17 10:22:50 dhcp209-217 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.submonitor_base.SubmonitorBase ERROR Error executing submonitor mgmt-bridge, args {'use_ssl': 'true', 'bridge_name': 'ovirtmgmt', 'address': '0'}#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitor_base.py", line 115, in _worker#012 self.action(self._options)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitors/mgmt_bridge.py", line 42, in action#012 logger=self._log#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/util.py", line 187, in connect_vdsm_json_rpc#012 requestQueue=requestQueue,#012 File "/usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py", line 248, in connect#012 responseQueue)#012 File "/usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py", line 234, in _create#012 lazy_start=False)#012 File "/usr/lib/python2.7/site-packages/yajsonrpc/stompreactor.py", line 629, in StandAloneRpcClient#012 client = StompClient(utils.create_connected_socket(host, port, sslctx),#012 File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 947, in create_connected_socket#012 sock = sslctx.wrapSocket(sock)#012 File "/usr/lib/python2.7/site-packages/vdsm/sslutils.py", line 101, in wrapSocket#012 ca_certs=self.ca_certs)#012 File "/usr/lib64/python2.7/ssl.py", line 936, in wrap_socket#012 ciphers=ciphers)#012 File "/usr/lib64/python2.7/ssl.py", line 547, in __init__#012 self._context.load_verify_locations(ca_certs)#012IOError: [Errno 2] No such file or directory May 17 10:22:51 dhcp209-217 imgbased: 2018-05-17 10:22:51,727 [ERROR] (MainThread) Failed to migrate etc#012Traceback (most recent call last):#012 File "/tmp/tmp.Z8koGnSp2N/usr/lib/python2.7/site-packages/imgbased/plugins/osupdater.py", line 119, in on_new_layer#012 check_nist_layout(imgbase, new_lv)#012 File "/tmp/tmp.Z8koGnSp2N/usr/lib/python2.7/site-packages/imgbased/plugins/osupdater.py", line 210, in check_nist_layout#012 v.create(t, paths[t]["size"], paths[t]["attach"])#012 File "/tmp/tmp.Z8koGnSp2N/usr/lib/python2.7/site-packages/imgbased/volume.py", line 62, in create#012 Rsync().sync(where + "/", mount.target.rstrip("/"))#012 File "/tmp/tmp.Z8koGnSp2N/usr/lib/python2.7/site-packages/imgbased/utils.py", line 1021, in sync#012 self._run(cmd)#012 File "/tmp/tmp.Z8koGnSp2N/usr/lib/python2.7/site-packages/imgbased/utils.py", line 999, in _run#012 call(cmd)#012 File "/tmp/tmp.Z8koGnSp2N/usr/lib/python2.7/site-packages/imgbased/utils.py", line 153, in call#012 return subprocess.check_output(*args, **kwargs).strip()#012 File "/usr/lib64/python2.7/subprocess.py", line 575, in check_output#012 raise CalledProcessError(retcode, cmd, output=output)#012CalledProcessError: Command '['rsync', '-pogAXlHrx', '-SWc', '--no-i-r', '-t', '--exclude', 'mnt.*/*', '/var/crash//', u'/tmp/mnt.A57da']' returned non-zero exit status 23 May 17 10:22:52 dhcp209-217 python: detected unhandled Python exception in '/tmp/tmp.Z8koGnSp2N/usr/lib/python2.7/site-packages/imgbased/__main__.py' May 17 10:22:53 dhcp209-217 abrt-server: Executable '/tmp/tmp.Z8koGnSp2N/usr/lib/python2.7/site-packages/imgbased/__main__.py' doesn't belong to any package and ProcessUnpackaged is set to 'no' May 17 10:22:53 dhcp209-217 abrt-server: 'post-create' on '/var/tmp/abrt/Python-2018-05-17-10:22:53-10216' exited with 1 May 17 10:22:53 dhcp209-217 abrt-server: Deleting problem directory '/var/tmp/abrt/Python-2018-05-17-10:22:53-10216' May 17 10:22:53 dhcp209-217 yum[9727]: Installed: redhat-virtualization-host-image-update-4.2-20180508.0.el7_5.noarch May 17 10:22:56 dhcp209-217 yum[9727]: Updated: libgcc-4.8.5-28.el7_5.1.x86_64 May 17 10:22:57 dhcp209-217 yum[9727]: Updated: libgomp-4.8.5-28.el7_5.1.x86_64 May 17 10:22:58 dhcp209-217 yum[9727]: Erased: redhat-virtualization-host-image-update-placeholder-4.1-11.0.el7.noarch May 17 10:23:07 dhcp209-217 systemd-logind: New session 512 of user root. May 17 10:23:07 dhcp209-217 systemd: Started Session 512 of user root. May 17 10:23:07 dhcp209-217 systemd: Starting Session 512 of user root. Connection to dhcp209-217.gsslab.pnq.redhat.com closed by remote host. Connection to dhcp209-217.gsslab.pnq.redhat.com closed.
Hey Yuval - Can you grab this, please? I can't reproduce. I'll take another run at it tomorrow, but I think you have a lot more time overlap with Ulhas
Surething, can you please attach /var/log/I'm based.log ?
Here is the failure: 2018-05-17 10:22:26,570 [DEBUG] (MainThread) Calling: (['rsync', '-pogAXlHrx', '-SWc', '--no-i-r', '-t', '--exclude', 'mnt.*/*', '/var/crash//', u'/tmp/mnt.A57da'],) {'close_fds': True, 'stderr': -2} 2018-05-17 10:22:50,228 [DEBUG] (MainThread) Exception! rsync: rsync_xal_set: lremovexattr(""/tmp/mnt.A57da/127.0.0.1-2018-05-12-19:01:12/.vmcore.2gqRdq"","security.selinux") failed: Permission denied (13) rsync: rsync_xal_set: lremovexattr(""/tmp/mnt.A57da/127.0.0.1-2018-05-12-19:01:12/.vmcore-dmesg.txt.t6zmb4"","security.selinux") failed: Permission denied (13) rsync: rsync_xal_set: lremovexattr(""/tmp/mnt.A57da/127.0.0.1-2018-05-14-07:01:15/.vmcore.irdZ8H"","security.selinux") failed: Permission denied (13) rsync: rsync_xal_set: lremovexattr(""/tmp/mnt.A57da/127.0.0.1-2018-05-14-07:01:15/.vmcore-dmesg.txt.fFg18o"","security.selinux") failed: Permission denied (13) rsync: rsync_xal_set: lremovexattr(""/tmp/mnt.A57da/127.0.0.1-2018-05-14-11:38:10/.vmcore.60L785"","security.selinux") failed: Permission denied (13) rsync: rsync_xal_set: lremovexattr(""/tmp/mnt.A57da/127.0.0.1-2018-05-14-11:38:10/.vmcore-dmesg.txt.WAJbJP"","security.selinux") failed: Permission denied (13)
I can't reproduce this, we're using rsync -X, so not sure why rsync tries to remove the security.selinux xattr - can you perhaps share `ls -lZR /var/crash` ?
Test version: From: rhvh-4.1-0.20180126.0+1 To: redhat-virtualization-host-4.2-20180530.1 (imgbased-1.0.17-0.1.el7ev.noarch) engine: 4.1.11.2-0.1.el7 Test steps: 1. Install rhvh-4.1-0.20180126.0+1 via GUI. 2. Register to engine 4.1 3. generated kdump by run #echo c > /proc/sysrq-trigger 4. After reboot. Create VMs on engine side (this step is must) 5. Try to upgrade host from the engine. Test result: Failed to upgrade Host from engine, detail info please check log Failed to install Host cshao_730. Yum Error in PREIN scriptlet in rpm package redhat-virtualization-host-image-update-4.2-20180530.1.el7_5.noarch. Failed to install Host cshao_730. Failed to execute stage 'Package installation': One or more elements within Yum transaction failed. So I have to assigned this bug.
Created attachment 1446209 [details] upgrade-fail
Can you please attach the imgbase log from the host ?
Nevermind, I got it, it really has nothing to do with this bug - you have local storage domain on the system, so the update fails, this check is done in the rpm's %pre section: From the ovirt-host-mgmt log file: Local storage domains were found on the same filesystem as / ! Please migrate the data to a new LV before upgrading, or you will lose the VMs 2018-05-31 03:34:57 ERROR otopi.plugins.otopi.packagers.yumpackager yumpackager.error:85 Yum Error in PREIN scriptlet in rpm package redhat-virtualization-host-image-update-4.2-20180530.1.el7_5.noarch
This is a known consequence of https://bugzilla.redhat.com/show_bug.cgi?id=1549630 Please remove the local storage domain and re-test
(In reply to Ryan Barry from comment #21) > This is a known consequence of > https://bugzilla.redhat.com/show_bug.cgi?id=1549630 > > Please remove the local storage domain and re-test After remove the local storage domain and re-test, upgrade can succeed. So the bug is fixed, change bug status to VERIFIED.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:1820
BZ<2>Jira Resync
sync2jira