Bug 1579141

Summary: [RHV4.2] unable to upgrade Host 4.1 (rhvh-4.1-0.20180126.0) to RHVH 4.2 (rhvh-4.2.3.0-0.20180508.0)
Product: Red Hat Enterprise Virtualization Manager Reporter: Ulhas Surse <usurse>
Component: imgbasedAssignee: Yuval Turgeman <yturgema>
Status: CLOSED ERRATA QA Contact: Yaning Wang <yaniwang>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 4.2.3CC: amashah, cshao, dfediuck, eheftman, glamb, gveitmic, huzhao, jcall, jcrumple, jiaczhan, mjankula, mkalinin, olim, peyu, qiyuan, rbarry, sirao, usurse, weiwang, yaniwang, yaoxu, ycui, yzhao
Target Milestone: ovirt-4.2.3-1Keywords: ZStream
Target Release: ---Flags: peyu: testing_plan_complete+
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: imgbased-1.0.17 Doc Type: Bug Fix
Doc Text:
Previously, vmcore files that were created by kdump were not relabelled by kdumpctl after a reboot. As a result, Rsync returned an error when trying to remove the security.selinux attribute while syncing unlabelled files, as this is forbidden by selinux. In this release, restorecon is run on the source directory before syncing it, which enables the files to be synchronized with the correct labels.
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-06-11 06:56:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Node RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1582433    
Attachments:
Description Flags
upgrade-fail none

Description Ulhas Surse 2018-05-17 04:56:00 UTC
Description of problem:
Unable to upgrade to RHVH rhvh-4.2.3.0-0.20180508.0

Version-Release number of selected component (if applicable):
RHV 4.2 
RHV H- rhvh-4.2.3.0-0.20180508.0

How reproducible:
Always

Steps to Reproduce:
1. Upgraded Manager to 4.2 successfully
2. Try to upgrade host from the manager. 
3. It failed with error: unable to upgrade the host. 
4. Also tried manually from command to update the package redhat-virtualization-host-image-update. error: 




# yum install redhat-virtualization-host-image-update-4.2-20180508.0.el7_5.noarch -y
Loaded plugins: imgbased-persist, package_upload, product-id, search-disabled-repos, subscription-manager
Resolving Dependencies
--> Running transaction check
---> Package redhat-virtualization-host-image-update.noarch 0:4.2-20180508.0.el7_5 will be installed
--> Finished Dependency Resolution


Dependencies Resolved

===================================================================================================================================================================
 Package                                                           Arch                             Version                  Repository                   Size
===================================================================================================================================================================
Installing:
 redhat-virtualization-host-image-update                           noarch                           4.2-20180508.0.el7_5     rhel-7-server-rhvh-4-rpms     613 M
Transaction Summary
====================================================================================================================================================================
Install  1 Package

Total download size: 613 M
Installed size: 612 M
Downloading packages:
redhat-virtualization-host-image-update-4.2-20180508.0.el7_5.noarch.rpm                                                                                                                 | 613 MB  00:00:35     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : redhat-virtualization-host-image-update-4.2-20180508.0.el7_5.noarch                                       1/1 
warning: %post(redhat-virtualization-host-image-update-4.2-20180508.0.el7_5.noarch) scriptlet failed, exit status 1
Non-fatal POSTIN scriptlet failure in rpm package redhat-virtualization-host-image-update-4.2-20180508.0.el7_5.noarch
Uploading Package Profile
  Verifying  : redhat-virtualization-host-image-update-4.2-20180508.0.el7_5.noarch                                       1/1 

Installed:
  redhat-virtualization-host-image-update.noarch 0:4.2-20180508.0.el7_5                                                                                                                                        

Complete!


=============================================

The layer is showing but can't boot as not in menu entry:

# nodectl info
layers: 
  rhvh-4.1-0.20180126.0: 
    rhvh-4.1-0.20180126.0+1
  rhvh-4.2.3.0-0.20180508.0: 
  rhvh-4.1-0.20180425.0: 
    rhvh-4.1-0.20180425.0+1
bootloader: 
  default: rhvh-4.1-0.20180425.0+1
  entries: 
    rhvh-4.1-0.20180126.0+1: 
      index: 1
      title: rhvh-4.1-0.20180126.0
      kernel: /boot/rhvh-4.1-0.20180126.0+1/vmlinuz-3.10.0-693.17.1.el7.x86_64
      args: "ro crashkernel=auto rd.lvm.lv=rhvh/rhvh-4.1-0.20180126.0+1 rd.lvm.lv=rhvh/swap rhgb quiet LANG=en_US.UTF-8 img.bootid=rhvh-4.1-0.20180126.0+1"
      initrd: /boot/rhvh-4.1-0.20180126.0+1/initramfs-3.10.0-693.17.1.el7.x86_64.img
      root: /dev/rhvh/rhvh-4.1-0.20180126.0+1
    rhvh-4.1-0.20180425.0+1: 
      index: 0
      title: rhvh-4.1-0.20180425.0
      kernel: /boot/rhvh-4.1-0.20180425.0+1/vmlinuz-3.10.0-862.el7.x86_64
      args: "ro crashkernel=auto rd.lvm.lv=rhvh/swap rd.lvm.lv=rhvh/rhvh-4.1-0.20180425.0+1 rhgb quiet LANG=en_US.UTF-8 img.bootid=rhvh-4.1-0.20180425.0+1"
      initrd: /boot/rhvh-4.1-0.20180425.0+1/initramfs-3.10.0-862.el7.x86_64.img
      root: /dev/rhvh/rhvh-4.1-0.20180425.0+1
current_layer: rhvh-4.1-0.20180425.0+1


================================================

/var/log/messages: 


May 17 09:42:53 dhcp210-242 python: ansible-yum Invoked with allow_downgrade=False name=['ovirt-host'] validate_certs=True list=None disable_gpg_check=False conf_file=None install_repoquery=True state=latest disablerepo=None update_cache=True disable_plugin=[] enablerepo=None exclude=None security=False update_only=False enable_plugin=[] installroot=/ skip_broken=False
May 17 09:43:07 dhcp210-242 python: ansible-yum Invoked with allow_downgrade=False name=['*'] validate_certs=True list=None disable_gpg_check=False conf_file=None install_repoquery=True state=latest disablerepo=None update_cache=True disable_plugin=[] enablerepo=None exclude=None security=False update_only=False enable_plugin=[] installroot=/ skip_broken=False
May 17 09:44:15 dhcp210-242 systemd-logind: Removed session 4.
May 17 09:44:36 dhcp210-242 kernel: loop: module loaded
May 17 09:44:37 dhcp210-242 kernel: squashfs: version 4.0 (2009/01/31) Phillip Lougher
May 17 09:44:37 dhcp210-242 kernel: EXT4-fs (loop1): mounted filesystem with ordered data mode. Opts: (null)
May 17 09:44:37 dhcp210-242 imgbased: 2018-05-17 09:44:37,320 [INFO] (MainThread) Extracting image '/usr/share/redhat-virtualization-host/image//redhat-virtualization-host-4.2-20180508.0.el7_5.squashfs.img'
May 17 09:44:37 dhcp210-242 kernel: EXT4-fs (loop3): mounted filesystem with ordered data mode. Opts: (null)
May 17 09:44:37 dhcp210-242 imgbased: 2018-05-17 09:44:37,497 [INFO] (MainThread) Starting base creation
May 17 09:44:37 dhcp210-242 imgbased: 2018-05-17 09:44:37,497 [INFO] (MainThread) New base will be: rhvh-4.2.3.0-0.20180508.0
May 17 09:44:37 dhcp210-242 python: detected unhandled Python exception in '/tmp/tmp.wib3Mcwa4l/usr/lib/python2.7/site-packages/imgbased/__main__.py'
May 17 09:44:38 dhcp210-242 abrt-server: Executable '/tmp/tmp.wib3Mcwa4l/usr/lib/python2.7/site-packages/imgbased/__main__.py' doesn't belong to any package and ProcessUnpackaged is set to 'no'
May 17 09:44:38 dhcp210-242 abrt-server: 'post-create' on '/var/tmp/abrt/Python-2018-05-17-09:44:37-22229' exited with 1
May 17 09:44:38 dhcp210-242 abrt-server: Deleting problem directory '/var/tmp/abrt/Python-2018-05-17-09:44:37-22229'
May 17 09:44:38 dhcp210-242 yum[21988]: Installed: redhat-virtualization-host-image-update-4.2-20180508.0.el7_5.noarch


===============================================


Actual results:
Failed to upgrade to 4.2 


Expected results:
Successfully upgrade the host. 

Additional info:

Comment 1 Ulhas Surse 2018-05-17 04:56:45 UTC
Tried with another node too: Failed with same error: 

node 2


May 17 10:16:28 dhcp209-217 python: ansible-yum Invoked with allow_downgrade=False name=['ovirt-host'] validate_certs=True list=None disable_gpg_check=False conf_file=None install_repoquery=True state=latest disablerepo=None update_cache=True disable_plugin=[] enablerepo=None exclude=None security=False update_only=False enable_plugin=[] installroot=/ skip_broken=False
May 17 10:16:50 dhcp209-217 python: ansible-yum Invoked with allow_downgrade=False name=['*'] validate_certs=True list=None disable_gpg_check=False conf_file=None install_repoquery=True state=latest disablerepo=None update_cache=True disable_plugin=[] enablerepo=None exclude=None security=False update_only=False enable_plugin=[] installroot=/ skip_broken=False
May 17 10:17:31 dhcp209-217 systemd: Started Session 510 of user root.
May 17 10:17:31 dhcp209-217 systemd-logind: New session 510 of user root.
May 17 10:17:31 dhcp209-217 systemd: Starting Session 510 of user root.






May 17 10:19:02 dhcp209-217 kernel: loop: module loaded
May 17 10:19:02 dhcp209-217 kernel: squashfs: version 4.0 (2009/01/31) Phillip Lougher
May 17 10:19:02 dhcp209-217 kernel: EXT4-fs (loop1): mounted filesystem with ordered data mode. Opts: (null)
May 17 10:19:02 dhcp209-217 imgbased: 2018-05-17 10:19:02,600 [INFO] (MainThread) Extracting image '/usr/share/redhat-virtualization-host/image//redhat-virtualization-host-4.2-20180508.0.el7_5.squashfs.img'
May 17 10:19:02 dhcp209-217 kernel: EXT4-fs (loop3): mounted filesystem with ordered data mode. Opts: (null)
May 17 10:19:03 dhcp209-217 imgbased: 2018-05-17 10:19:03,128 [INFO] (MainThread) Starting base creation
May 17 10:19:03 dhcp209-217 imgbased: 2018-05-17 10:19:03,128 [INFO] (MainThread) New base will be: rhvh-4.2.3.0-0.20180508.0
May 17 10:19:04 dhcp209-217 imgbased: 2018-05-17 10:19:04,059 [INFO] (MainThread) New LV is: <LV 'rhvh/rhvh-4.2.3.0-0.20180508.0' />
May 17 10:19:04 dhcp209-217 dmeventd[824]: No longer monitoring thin pool rhvh-pool00-tpool.
May 17 10:19:04 dhcp209-217 lvm[824]: Monitoring thin pool rhvh-pool00-tpool.
May 17 10:19:04 dhcp209-217 multipathd: dm-13: remove map (uevent)
May 17 10:19:04 dhcp209-217 multipathd: dm-13: devmap not registered, can't remove
May 17 10:19:04 dhcp209-217 multipathd: dm-13: remove map (uevent)
May 17 10:19:05 dhcp209-217 imgbased: 2018-05-17 10:19:05,420 [INFO] (MainThread) Creating new filesystem on base
May 17 10:19:08 dhcp209-217 imgbased: 2018-05-17 10:19:08,437 [INFO] (MainThread) Writing tree to base
May 17 10:19:08 dhcp209-217 kernel: EXT4-fs (dm-13): mounted filesystem with ordered data mode. Opts: (null)
May 17 10:20:02 dhcp209-217 systemd: Started Session 511 of user root.
May 17 10:20:02 dhcp209-217 systemd: Starting Session 511 of user root.
May 17 10:21:43 dhcp209-217 kernel: SELinux:  Context system_u:object_r:ovirt_vmconsole_exec_t:s0 is not valid (left unmapped).
May 17 10:21:54 dhcp209-217 dmeventd[824]: No longer monitoring thin pool rhvh-pool00-tpool.
May 17 10:21:54 dhcp209-217 lvm[824]: Monitoring thin pool rhvh-pool00-tpool.
May 17 10:21:55 dhcp209-217 multipathd: dm-13: remove map (uevent)
May 17 10:21:55 dhcp209-217 multipathd: dm-13: devmap not registered, can't remove
May 17 10:21:55 dhcp209-217 multipathd: dm-13: remove map (uevent)
May 17 10:21:55 dhcp209-217 imgbased: 2018-05-17 10:21:55,428 [INFO] (MainThread) Adding a new layer after <Base rhvh-4.2.3.0-0.20180508.0 [] />
May 17 10:21:55 dhcp209-217 imgbased: 2018-05-17 10:21:55,428 [INFO] (MainThread) Adding a new layer after <Base rhvh-4.2.3.0-0.20180508.0 [] />
May 17 10:21:55 dhcp209-217 imgbased: 2018-05-17 10:21:55,429 [INFO] (MainThread) New layer will be: <Layer rhvh-4.2.3.0-0.20180508.0+1 />
May 17 10:21:55 dhcp209-217 dmeventd[824]: No longer monitoring thin pool rhvh-pool00-tpool.
May 17 10:21:55 dhcp209-217 lvm[824]: Monitoring thin pool rhvh-pool00-tpool.
May 17 10:21:58 dhcp209-217 kernel: EXT4-fs (dm-14): mounted filesystem with ordered data mode. Opts: (null)
May 17 10:21:58 dhcp209-217 kernel: EXT4-fs (dm-14): mounted filesystem with ordered data mode. Opts: (null)
May 17 10:21:59 dhcp209-217 kernel: EXT4-fs (dm-14): mounted filesystem with ordered data mode. Opts: (null)
May 17 10:22:00 dhcp209-217 kernel: EXT4-fs (dm-12): mounted filesystem with ordered data mode. Opts: (null)
May 17 10:22:05 dhcp209-217 dbus[1006]: [system] Reloaded configuration
May 17 10:22:13 dhcp209-217 imgbased: 2018-05-17 10:22:13,213 [INFO] (remediate_etc) Syncing systemd levels
May 17 10:22:13 dhcp209-217 dbus[1006]: [system] Reloaded configuration
May 17 10:22:14 dhcp209-217 kernel: EXT4-fs (dm-14): mounted filesystem with ordered data mode. Opts: (null)
May 17 10:22:21 dhcp209-217 imgbased: 2018-05-17 10:22:21,571 [INFO] (remediate_etc) Syncing systemd levels
May 17 10:22:22 dhcp209-217 kernel: EXT4-fs (dm-14): mounted filesystem with ordered data mode. Opts: (null)
May 17 10:22:22 dhcp209-217 kernel: device-mapper: thin: Data device (dm-2) discard unsupported: Disabling discard passdown.
May 17 10:22:26 dhcp209-217 kernel: EXT4-fs (dm-15): mounted filesystem with ordered data mode. Opts: (null)
May 17 10:22:29 dhcp209-217 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.submonitor_base.SubmonitorBase ERROR Error executing submonitor engine-health, args {'use_ssl': 'true', 'vm_uuid': '5a1dab2d-c19f-4e51-a225-71882c535084', 'address': '0'}#012Traceback (most recent call last):#012  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitor_base.py", line 115, in _worker#012    self.action(self._options)#012  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitors/engine_health.py", line 49, in action#012    logger=self._log#012  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/util.py", line 187, in connect_vdsm_json_rpc#012    requestQueue=requestQueue,#012  File "/usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py", line 248, in connect#012    responseQueue)#012  File "/usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py", line 234, in _create#012    lazy_start=False)#012  File "/usr/lib/python2.7/site-packages/yajsonrpc/stompreactor.py", line 629, in StandAloneRpcClient#012    client = StompClient(utils.create_connected_socket(host, port, sslctx),#012  File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 947, in create_connected_socket#012    sock = sslctx.wrapSocket(sock)#012  File "/usr/lib/python2.7/site-packages/vdsm/sslutils.py", line 101, in wrapSocket#012    ca_certs=self.ca_certs)#012  File "/usr/lib64/python2.7/ssl.py", line 936, in wrap_socket#012    ciphers=ciphers)#012  File "/usr/lib64/python2.7/ssl.py", line 547, in __init__#012    self._context.load_verify_locations(ca_certs)#012IOError: [Errno 2] No such file or directory
May 17 10:22:29 dhcp209-217 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.submonitor_base.SubmonitorBase ERROR Error executing submonitor mem-free, args {'use_ssl': 'true', 'address': '0'}#012Traceback (most recent call last):#012  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitor_base.py", line 115, in _worker#012    self.action(self._options)#012  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitors/mem_free.py", line 38, in action#012    logger=self._log#012  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/util.py", line 187, in connect_vdsm_json_rpc#012    requestQueue=requestQueue,#012  File "/usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py", line 248, in connect#012    responseQueue)#012  File "/usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py", line 234, in _create#012    lazy_start=False)#012  File "/usr/lib/python2.7/site-packages/yajsonrpc/stompreactor.py", line 629, in StandAloneRpcClient#012    client = StompClient(utils.create_connected_socket(host, port, sslctx),#012  File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 947, in create_connected_socket#012    sock = sslctx.wrapSocket(sock)#012  File "/usr/lib/python2.7/site-packages/vdsm/sslutils.py", line 101, in wrapSocket#012    ca_certs=self.ca_certs)#012  File "/usr/lib64/python2.7/ssl.py", line 936, in wrap_socket#012    ciphers=ciphers)#012  File "/usr/lib64/python2.7/ssl.py", line 547, in __init__#012    self._context.load_verify_locations(ca_certs)#012IOError: [Errno 2] No such file or directory
May 17 10:22:30 dhcp209-217 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.submonitor_base.SubmonitorBase ERROR Error executing submonitor mgmt-bridge, args {'use_ssl': 'true', 'bridge_name': 'ovirtmgmt', 'address': '0'}#012Traceback (most recent call last):#012  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitor_base.py", line 115, in _worker#012    self.action(self._options)#012  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitors/mgmt_bridge.py", line 42, in action#012    logger=self._log#012  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/util.py", line 187, in connect_vdsm_json_rpc#012    requestQueue=requestQueue,#012  File "/usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py", line 248, in connect#012    responseQueue)#012  File "/usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py", line 234, in _create#012    lazy_start=False)#012  File "/usr/lib/python2.7/site-packages/yajsonrpc/stompreactor.py", line 629, in StandAloneRpcClient#012    client = StompClient(utils.create_connected_socket(host, port, sslctx),#012  File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 947, in create_connected_socket#012    sock = sslctx.wrapSocket(sock)#012  File "/usr/lib/python2.7/site-packages/vdsm/sslutils.py", line 101, in wrapSocket#012    ca_certs=self.ca_certs)#012  File "/usr/lib64/python2.7/ssl.py", line 936, in wrap_socket#012    ciphers=ciphers)#012  File "/usr/lib64/python2.7/ssl.py", line 547, in __init__#012    self._context.load_verify_locations(ca_certs)#012IOError: [Errno 2] No such file or directory
May 17 10:22:38 dhcp209-217 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.submonitor_base.SubmonitorBase ERROR Error executing submonitor engine-health, args {'use_ssl': 'true', 'vm_uuid': '5a1dab2d-c19f-4e51-a225-71882c535084', 'address': '0'}#012Traceback (most recent call last):#012  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitor_base.py", line 115, in _worker#012    self.action(self._options)#012  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitors/engine_health.py", line 49, in action#012    logger=self._log#012  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/util.py", line 187, in connect_vdsm_json_rpc#012    requestQueue=requestQueue,#012  File "/usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py", line 248, in connect#012    responseQueue)#012  File "/usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py", line 234, in _create#012    lazy_start=False)#012  File "/usr/lib/python2.7/site-packages/yajsonrpc/stompreactor.py", line 629, in StandAloneRpcClient#012    client = StompClient(utils.create_connected_socket(host, port, sslctx),#012  File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 947, in create_connected_socket#012    sock = sslctx.wrapSocket(sock)#012  File "/usr/lib/python2.7/site-packages/vdsm/sslutils.py", line 101, in wrapSocket#012    ca_certs=self.ca_certs)#012  File "/usr/lib64/python2.7/ssl.py", line 936, in wrap_socket#012    ciphers=ciphers)#012  File "/usr/lib64/python2.7/ssl.py", line 547, in __init__#012    self._context.load_verify_locations(ca_certs)#012IOError: [Errno 2] No such file or directory
May 17 10:22:38 dhcp209-217 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.submonitor_base.SubmonitorBase ERROR Error executing submonitor mem-free, args {'use_ssl': 'true', 'address': '0'}#012Traceback (most recent call last):#012  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitor_base.py", line 115, in _worker#012    self.action(self._options)#012  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitors/mem_free.py", line 38, in action#012    logger=self._log#012  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/util.py", line 187, in connect_vdsm_json_rpc#012    requestQueue=requestQueue,#012  File "/usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py", line 248, in connect#012    responseQueue)#012  File "/usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py", line 234, in _create#012    lazy_start=False)#012  File "/usr/lib/python2.7/site-packages/yajsonrpc/stompreactor.py", line 629, in StandAloneRpcClient#012    client = StompClient(utils.create_connected_socket(host, port, sslctx),#012  File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 947, in create_connected_socket#012    sock = sslctx.wrapSocket(sock)#012  File "/usr/lib/python2.7/site-packages/vdsm/sslutils.py", line 101, in wrapSocket#012    ca_certs=self.ca_certs)#012  File "/usr/lib64/python2.7/ssl.py", line 936, in wrap_socket#012    ciphers=ciphers)#012  File "/usr/lib64/python2.7/ssl.py", line 547, in __init__#012    self._context.load_verify_locations(ca_certs)#012IOError: [Errno 2] No such file or directory
May 17 10:22:40 dhcp209-217 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.submonitor_base.SubmonitorBase ERROR Error executing submonitor mgmt-bridge, args {'use_ssl': 'true', 'bridge_name': 'ovirtmgmt', 'address': '0'}#012Traceback (most recent call last):#012  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitor_base.py", line 115, in _worker#012    self.action(self._options)#012  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitors/mgmt_bridge.py", line 42, in action#012    logger=self._log#012  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/util.py", line 187, in connect_vdsm_json_rpc#012    requestQueue=requestQueue,#012  File "/usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py", line 248, in connect#012    responseQueue)#012  File "/usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py", line 234, in _create#012    lazy_start=False)#012  File "/usr/lib/python2.7/site-packages/yajsonrpc/stompreactor.py", line 629, in StandAloneRpcClient#012    client = StompClient(utils.create_connected_socket(host, port, sslctx),#012  File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 947, in create_connected_socket#012    sock = sslctx.wrapSocket(sock)#012  File "/usr/lib/python2.7/site-packages/vdsm/sslutils.py", line 101, in wrapSocket#012    ca_certs=self.ca_certs)#012  File "/usr/lib64/python2.7/ssl.py", line 936, in wrap_socket#012    ciphers=ciphers)#012  File "/usr/lib64/python2.7/ssl.py", line 547, in __init__#012    self._context.load_verify_locations(ca_certs)#012IOError: [Errno 2] No such file or directory
May 17 10:22:48 dhcp209-217 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.submonitor_base.SubmonitorBase ERROR Error executing submonitor engine-health, args {'use_ssl': 'true', 'vm_uuid': '5a1dab2d-c19f-4e51-a225-71882c535084', 'address': '0'}#012Traceback (most recent call last):#012  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitor_base.py", line 115, in _worker#012    self.action(self._options)#012  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitors/engine_health.py", line 49, in action#012    logger=self._log#012  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/util.py", line 187, in connect_vdsm_json_rpc#012    requestQueue=requestQueue,#012  File "/usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py", line 248, in connect#012    responseQueue)#012  File "/usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py", line 234, in _create#012    lazy_start=False)#012  File "/usr/lib/python2.7/site-packages/yajsonrpc/stompreactor.py", line 629, in StandAloneRpcClient#012    client = StompClient(utils.create_connected_socket(host, port, sslctx),#012  File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 947, in create_connected_socket#012    sock = sslctx.wrapSocket(sock)#012  File "/usr/lib/python2.7/site-packages/vdsm/sslutils.py", line 101, in wrapSocket#012    ca_certs=self.ca_certs)#012  File "/usr/lib64/python2.7/ssl.py", line 936, in wrap_socket#012    ciphers=ciphers)#012  File "/usr/lib64/python2.7/ssl.py", line 547, in __init__#012    self._context.load_verify_locations(ca_certs)#012IOError: [Errno 2] No such file or directory
May 17 10:22:48 dhcp209-217 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.submonitor_base.SubmonitorBase ERROR Error executing submonitor mem-free, args {'use_ssl': 'true', 'address': '0'}#012Traceback (most recent call last):#012  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitor_base.py", line 115, in _worker#012    self.action(self._options)#012  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitors/mem_free.py", line 38, in action#012    logger=self._log#012  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/util.py", line 187, in connect_vdsm_json_rpc#012    requestQueue=requestQueue,#012  File "/usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py", line 248, in connect#012    responseQueue)#012  File "/usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py", line 234, in _create#012    lazy_start=False)#012  File "/usr/lib/python2.7/site-packages/yajsonrpc/stompreactor.py", line 629, in StandAloneRpcClient#012    client = StompClient(utils.create_connected_socket(host, port, sslctx),#012  File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 947, in create_connected_socket#012    sock = sslctx.wrapSocket(sock)#012  File "/usr/lib/python2.7/site-packages/vdsm/sslutils.py", line 101, in wrapSocket#012    ca_certs=self.ca_certs)#012  File "/usr/lib64/python2.7/ssl.py", line 936, in wrap_socket#012    ciphers=ciphers)#012  File "/usr/lib64/python2.7/ssl.py", line 547, in __init__#012    self._context.load_verify_locations(ca_certs)#012IOError: [Errno 2] No such file or directory
May 17 10:22:50 dhcp209-217 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.submonitor_base.SubmonitorBase ERROR Error executing submonitor mgmt-bridge, args {'use_ssl': 'true', 'bridge_name': 'ovirtmgmt', 'address': '0'}#012Traceback (most recent call last):#012  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitor_base.py", line 115, in _worker#012    self.action(self._options)#012  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitors/mgmt_bridge.py", line 42, in action#012    logger=self._log#012  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/util.py", line 187, in connect_vdsm_json_rpc#012    requestQueue=requestQueue,#012  File "/usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py", line 248, in connect#012    responseQueue)#012  File "/usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py", line 234, in _create#012    lazy_start=False)#012  File "/usr/lib/python2.7/site-packages/yajsonrpc/stompreactor.py", line 629, in StandAloneRpcClient#012    client = StompClient(utils.create_connected_socket(host, port, sslctx),#012  File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 947, in create_connected_socket#012    sock = sslctx.wrapSocket(sock)#012  File "/usr/lib/python2.7/site-packages/vdsm/sslutils.py", line 101, in wrapSocket#012    ca_certs=self.ca_certs)#012  File "/usr/lib64/python2.7/ssl.py", line 936, in wrap_socket#012    ciphers=ciphers)#012  File "/usr/lib64/python2.7/ssl.py", line 547, in __init__#012    self._context.load_verify_locations(ca_certs)#012IOError: [Errno 2] No such file or directory
May 17 10:22:51 dhcp209-217 imgbased: 2018-05-17 10:22:51,727 [ERROR] (MainThread) Failed to migrate etc#012Traceback (most recent call last):#012  File "/tmp/tmp.Z8koGnSp2N/usr/lib/python2.7/site-packages/imgbased/plugins/osupdater.py", line 119, in on_new_layer#012    check_nist_layout(imgbase, new_lv)#012  File "/tmp/tmp.Z8koGnSp2N/usr/lib/python2.7/site-packages/imgbased/plugins/osupdater.py", line 210, in check_nist_layout#012    v.create(t, paths[t]["size"], paths[t]["attach"])#012  File "/tmp/tmp.Z8koGnSp2N/usr/lib/python2.7/site-packages/imgbased/volume.py", line 62, in create#012    Rsync().sync(where + "/", mount.target.rstrip("/"))#012  File "/tmp/tmp.Z8koGnSp2N/usr/lib/python2.7/site-packages/imgbased/utils.py", line 1021, in sync#012    self._run(cmd)#012  File "/tmp/tmp.Z8koGnSp2N/usr/lib/python2.7/site-packages/imgbased/utils.py", line 999, in _run#012    call(cmd)#012  File "/tmp/tmp.Z8koGnSp2N/usr/lib/python2.7/site-packages/imgbased/utils.py", line 153, in call#012    return subprocess.check_output(*args, **kwargs).strip()#012  File "/usr/lib64/python2.7/subprocess.py", line 575, in check_output#012    raise CalledProcessError(retcode, cmd, output=output)#012CalledProcessError: Command '['rsync', '-pogAXlHrx', '-SWc', '--no-i-r', '-t', '--exclude', 'mnt.*/*', '/var/crash//', u'/tmp/mnt.A57da']' returned non-zero exit status 23
May 17 10:22:52 dhcp209-217 python: detected unhandled Python exception in '/tmp/tmp.Z8koGnSp2N/usr/lib/python2.7/site-packages/imgbased/__main__.py'
May 17 10:22:53 dhcp209-217 abrt-server: Executable '/tmp/tmp.Z8koGnSp2N/usr/lib/python2.7/site-packages/imgbased/__main__.py' doesn't belong to any package and ProcessUnpackaged is set to 'no'
May 17 10:22:53 dhcp209-217 abrt-server: 'post-create' on '/var/tmp/abrt/Python-2018-05-17-10:22:53-10216' exited with 1
May 17 10:22:53 dhcp209-217 abrt-server: Deleting problem directory '/var/tmp/abrt/Python-2018-05-17-10:22:53-10216'
May 17 10:22:53 dhcp209-217 yum[9727]: Installed: redhat-virtualization-host-image-update-4.2-20180508.0.el7_5.noarch
May 17 10:22:56 dhcp209-217 yum[9727]: Updated: libgcc-4.8.5-28.el7_5.1.x86_64
May 17 10:22:57 dhcp209-217 yum[9727]: Updated: libgomp-4.8.5-28.el7_5.1.x86_64
May 17 10:22:58 dhcp209-217 yum[9727]: Erased: redhat-virtualization-host-image-update-placeholder-4.1-11.0.el7.noarch
May 17 10:23:07 dhcp209-217 systemd-logind: New session 512 of user root.
May 17 10:23:07 dhcp209-217 systemd: Started Session 512 of user root.
May 17 10:23:07 dhcp209-217 systemd: Starting Session 512 of user root.
Connection to dhcp209-217.gsslab.pnq.redhat.com closed by remote host.
Connection to dhcp209-217.gsslab.pnq.redhat.com closed.

Comment 3 Ryan Barry 2018-05-17 19:23:15 UTC
Hey Yuval -

Can you grab this, please? I can't reproduce. I'll take another run at it tomorrow, but I think you have a lot more time overlap with Ulhas

Comment 4 Yuval Turgeman 2018-05-17 19:37:48 UTC
Surething, can you please attach /var/log/I'm based.log ?

Comment 5 Germano Veit Michel 2018-05-18 01:25:21 UTC
Here is the failure:

2018-05-17 10:22:26,570 [DEBUG] (MainThread) Calling: (['rsync', '-pogAXlHrx', '-SWc', '--no-i-r', '-t', '--exclude', 'mnt.*/*', '/var/crash//', u'/tmp/mnt.A57da'],) {'close_fds': True, 'stderr': -2}
2018-05-17 10:22:50,228 [DEBUG] (MainThread) Exception! rsync: rsync_xal_set: lremovexattr(""/tmp/mnt.A57da/127.0.0.1-2018-05-12-19:01:12/.vmcore.2gqRdq"","security.selinux") failed: Permission denied (13)
rsync: rsync_xal_set: lremovexattr(""/tmp/mnt.A57da/127.0.0.1-2018-05-12-19:01:12/.vmcore-dmesg.txt.t6zmb4"","security.selinux") failed: Permission denied (13)
rsync: rsync_xal_set: lremovexattr(""/tmp/mnt.A57da/127.0.0.1-2018-05-14-07:01:15/.vmcore.irdZ8H"","security.selinux") failed: Permission denied (13)
rsync: rsync_xal_set: lremovexattr(""/tmp/mnt.A57da/127.0.0.1-2018-05-14-07:01:15/.vmcore-dmesg.txt.fFg18o"","security.selinux") failed: Permission denied (13)
rsync: rsync_xal_set: lremovexattr(""/tmp/mnt.A57da/127.0.0.1-2018-05-14-11:38:10/.vmcore.60L785"","security.selinux") failed: Permission denied (13)
rsync: rsync_xal_set: lremovexattr(""/tmp/mnt.A57da/127.0.0.1-2018-05-14-11:38:10/.vmcore-dmesg.txt.WAJbJP"","security.selinux") failed: Permission denied (13)

Comment 8 Yuval Turgeman 2018-05-21 11:19:29 UTC
I can't reproduce this, we're using rsync -X, so not sure why rsync tries to remove the security.selinux xattr - can you perhaps share `ls -lZR /var/crash` ?

Comment 17 cshao 2018-05-31 10:05:52 UTC
Test version:
From: rhvh-4.1-0.20180126.0+1
To: redhat-virtualization-host-4.2-20180530.1 (imgbased-1.0.17-0.1.el7ev.noarch)
engine: 4.1.11.2-0.1.el7


Test steps:
1. Install rhvh-4.1-0.20180126.0+1 via GUI.
2. Register to engine 4.1
3. generated kdump by run 
   #echo c > /proc/sysrq-trigger
4. After reboot. Create VMs on engine side (this step is must)
5. Try to upgrade host from the engine. 

Test result:
Failed to upgrade Host from engine, detail info please check log

Failed to install Host cshao_730. Yum Error in PREIN scriptlet in rpm package redhat-virtualization-host-image-update-4.2-20180530.1.el7_5.noarch.
	
Failed to install Host cshao_730. Failed to execute stage 'Package installation': One or more elements within Yum transaction failed.


So I have to assigned this bug.

Comment 18 cshao 2018-05-31 10:07:02 UTC
Created attachment 1446209 [details]
upgrade-fail

Comment 19 Yuval Turgeman 2018-05-31 10:53:17 UTC
Can you please attach the imgbase log from the host ?

Comment 20 Yuval Turgeman 2018-05-31 11:42:39 UTC
Nevermind, I got it, it really has nothing to do with this bug - you have local storage domain on the system, so the update fails, this check is done in the rpm's %pre section:

From the ovirt-host-mgmt log file:

Local storage domains were found on the same filesystem as / ! Please migrate the data to a new LV before upgrading, or you will lose the VMs

2018-05-31 03:34:57 ERROR otopi.plugins.otopi.packagers.yumpackager yumpackager.error:85 Yum Error in PREIN scriptlet in rpm package redhat-virtualization-host-image-update-4.2-20180530.1.el7_5.noarch

Comment 21 Ryan Barry 2018-05-31 11:49:58 UTC
This is a known consequence of https://bugzilla.redhat.com/show_bug.cgi?id=1549630

Please remove the local storage domain and re-test

Comment 22 cshao 2018-06-01 08:02:25 UTC
(In reply to Ryan Barry from comment #21)
> This is a known consequence of
> https://bugzilla.redhat.com/show_bug.cgi?id=1549630
> 
> Please remove the local storage domain and re-test

After remove the local storage domain and re-test, upgrade can succeed.
So the bug is fixed, change bug status to VERIFIED.

Comment 32 errata-xmlrpc 2018-06-11 06:56:53 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:1820

Comment 33 Franta Kust 2019-05-16 13:09:07 UTC
BZ<2>Jira Resync

Comment 34 Daniel Gur 2019-08-28 13:14:58 UTC
sync2jira

Comment 35 Daniel Gur 2019-08-28 13:20:01 UTC
sync2jira