Bug 1476094 - Upgrade failed when upgrade to rhvh-4.1-20170727.1
Upgrade failed when upgrade to rhvh-4.1-20170727.1
Status: CLOSED CURRENTRELEASE
Product: ovirt-node
Classification: oVirt
Component: Installation & Update (Show other bugs)
4.1
Unspecified Unspecified
unspecified Severity urgent (vote)
: ovirt-4.1.4-1
: ---
Assigned To: Ryan Barry
Huijuan Zhao
: Rebase, Regression, TestBlocker
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-07-27 23:16 EDT by Huijuan Zhao
Modified: 2017-08-23 04:00 EDT (History)
13 users (show)

See Also:
Fixed In Version: imgbased-0.9.36-0.1.el7ev
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-08-23 04:00:41 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Node
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
rule-engine: ovirt‑4.1?
rule-engine: blocker?
rbarry: planning_ack?
rbarry: devel_ack+
cshao: testing_ack+


Attachments (Terms of Use)
Sosreport, and all logs in /var/log and /tmp from host (8.52 MB, application/x-gzip)
2017-07-27 23:16 EDT, Huijuan Zhao
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 79931 master MERGED utils: use lazy umounting 2017-07-28 07:59 EDT
oVirt gerrit 79932 ovirt-4.1 MERGED utils: use lazy umounting 2017-07-28 08:00 EDT

  None (edit)
Description Huijuan Zhao 2017-07-27 23:16:27 EDT
Created attachment 1305767 [details]
Sosreport, and all logs in /var/log and /tmp from host

Description of problem:
Upgrade from rhvh-4.0 to rhvh-4.1-20170727.1, there is error info during upgrade, and missing boot entry of rhvh-4.1-20170727.1 after upgrade.


Version-Release number of selected component (if applicable):
From: redhat-virtualization-host-4.0-20170307.1
To:   redhat-virtualization-host-4.1-20170727.1
      

How reproducible:
100%


Steps to Reproduce:
1. Clean install redhat-virtualization-host-4.0-20170307.1
2. Setup local repo in rhvh-4.0, and upgrade to redhat-virtualization-host-4.1-20170727.1
   # yum upgrade
3. Focus on the upgrade process
4. Reboot rhvh after upgrade, and focus on the boot entry


Actual results:
1. After step 3, there is error info during upgrade:

# yum update
Loaded plugins: imgbased-warning, product-id, search-disabled-repos, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Warning: yum operations are not persisted across upgrades!
Repo rhel-7-server-rhvh-4-source-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877.pem
Repo rhel-7-server-rhvh-4-source-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877-key.pem
Repo rhel-7-server-rhvh-4-beta-debug-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877.pem
Repo rhel-7-server-rhvh-4-beta-debug-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877-key.pem
Repo rhel-7-server-rhvh-4-debug-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877.pem
Repo rhel-7-server-rhvh-4-debug-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877-key.pem
Repo rhel-7-server-rhvh-4-beta-source-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877.pem
Repo rhel-7-server-rhvh-4-beta-source-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877-key.pem
Repo rhel-7-server-rhvh-4-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877.pem
Repo rhel-7-server-rhvh-4-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877-key.pem
Repo rhel-7-server-rhvh-4-beta-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877.pem
Repo rhel-7-server-rhvh-4-beta-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877-key.pem
rhel-7-server-rhvh-4-beta-debug-rpms                                                                                                           | 2.9 kB  00:00:00    
rhel-7-server-rhvh-4-beta-rpms                                                                                                                 | 2.9 kB  00:00:00    
rhel-7-server-rhvh-4-beta-source-rpms                                                                                                          | 2.9 kB  00:00:00    
rhel-7-server-rhvh-4-debug-rpms                                                                                                                | 2.9 kB  00:00:00    
rhel-7-server-rhvh-4-rpms                                                                                                                      | 2.9 kB  00:00:00    
rhel-7-server-rhvh-4-source-rpms                                                                                                               | 2.9 kB  00:00:00    
(1/6): rhel-7-server-rhvh-4-beta-debug-rpms/primary_db                                                                                         | 1.7 kB  00:00:00    
(2/6): rhel-7-server-rhvh-4-source-rpms/primary_db                                                                                             | 4.1 kB  00:00:00    
(3/6): rhel-7-server-rhvh-4-debug-rpms/primary_db                                                                                              | 1.7 kB  00:00:00    
(4/6): rhel-7-server-rhvh-4-beta-source-rpms/primary_db                                                                                        | 4.1 kB  00:00:00    
(5/6): rhel-7-server-rhvh-4-beta-rpms/primary_db                                                                                               | 9.4 kB  00:00:00    
(6/6): rhel-7-server-rhvh-4-rpms/primary_db                                                                                                    | 9.4 kB  00:00:00    
Resolving Dependencies
--> Running transaction check
---> Package redhat-virtualization-host-image-update.noarch 0:4.1-20170727.1.el7_4 will be obsoleting
---> Package redhat-virtualization-host-image-update-placeholder.noarch 0:4.0-7.1.el7 will be obsoleted
--> Finished Dependency Resolution

Dependencies Resolved

======================================================================================================================================================================
 Package                                                Arch                  Version                             Repository                                     Size
======================================================================================================================================================================
Installing:
 redhat-virtualization-host-image-update                noarch                4.1-20170727.1.el7_4                rhel-7-server-rhvh-4-beta-rpms                603 M
     replacing  redhat-virtualization-host-image-update-placeholder.noarch 4.0-7.1.el7

Transaction Summary
======================================================================================================================================================================
Install  1 Package

Total download size: 603 M
Is this ok [y/d/N]: y
Downloading packages:
redhat-virtualization-host-image-update-latest.rpm                                                                                             | 603 MB  00:00:12    
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : redhat-virtualization-host-image-update-4.1-20170727.1.el7_4.noarch                                                                                1/2
Traceback (most recent call last):
  File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/tmp/tmp.mNS01rM13z/usr/lib/python2.7/site-packages/imgbased/__main__.py", line 53, in <module>
    CliApplication()
  File "/tmp/tmp.mNS01rM13z/usr/lib/python2.7/site-packages/imgbased/__init__.py", line 82, in CliApplication
    app.hooks.emit("post-arg-parse", args)
  File "/tmp/tmp.mNS01rM13z/usr/lib/python2.7/site-packages/imgbased/hooks.py", line 120, in emit
    cb(self.context, *args)
  File "/tmp/tmp.mNS01rM13z/usr/lib/python2.7/site-packages/imgbased/plugins/core.py", line 160, in post_argparse
    print(app.imgbase.latest_layer())
  File "/tmp/tmp.mNS01rM13z/usr/lib/python2.7/site-packages/imgbased/imgbase.py", line 373, in latest_layer
    return self.naming.last_layer()
  File "/tmp/tmp.mNS01rM13z/usr/lib/python2.7/site-packages/imgbased/naming.py", line 67, in last_layer
    layers = self.layers()
  File "/tmp/tmp.mNS01rM13z/usr/lib/python2.7/site-packages/imgbased/naming.py", line 56, in layers
    for b in self.tree():
  File "/tmp/tmp.mNS01rM13z/usr/lib/python2.7/site-packages/imgbased/naming.py", line 205, in tree
    names = datasource()
  File "/tmp/tmp.mNS01rM13z/usr/lib/python2.7/site-packages/imgbased/imgbase.py", line 99, in list_our_lv_names
    lvs = LVM.list_lvs(filtr=filtr)
  File "/tmp/tmp.mNS01rM13z/usr/lib/python2.7/site-packages/imgbased/lvm.py", line 63, in list_lvs
    lvs = [cls.LV.from_lvm_name(n) for n in cls._list_lv_full_names(filtr)]
  File "/tmp/tmp.mNS01rM13z/usr/lib/python2.7/site-packages/imgbased/lvm.py", line 56, in _list_lv_full_names
    raw = LVM._lvs(cmd)
  File "/tmp/tmp.mNS01rM13z/usr/lib/python2.7/site-packages/imgbased/utils.py", line 371, in lvs
    return self.call(["lvs"] + args, **kwargs)
  File "/tmp/tmp.mNS01rM13z/usr/lib/python2.7/site-packages/imgbased/utils.py", line 441, in call
    return super(LvmBinary, self).call(*args, stderr=DEVNULL, **kwargs)
  File "/tmp/tmp.mNS01rM13z/usr/lib/python2.7/site-packages/imgbased/utils.py", line 365, in call
    stdout = call(*args, **kwargs)
  File "/tmp/tmp.mNS01rM13z/usr/lib/python2.7/site-packages/imgbased/utils.py", line 147, in call
    return subprocess.check_output(*args, **kwargs).strip()
  File "/usr/lib64/python2.7/subprocess.py", line 575, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['lvs', '--noheadings', '-o', 'lv_full_name', '--select', 'lv_tags = imgbased:base || lv_tags = imgbased:layer']' returned non-zero exit status 255
usage: imgbase layer [-h] [--add] [--latest] [--current] [--volume-path NVR]
                     [IMAGE]
imgbase layer: error: argument --volume-path: expected one argument
mount:  is write-protected, mounting read-only
mount: wrong fs type, bad option, bad superblock on ,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
rm: cannot remove ‘/tmp/tmp.mNS01rM13z/usr/share/imgbased/*image-update*.rpm’: No such file or directory
umount: /tmp/tmp.mNS01rM13z: not mounted
  Erasing    : redhat-virtualization-host-image-update-placeholder-4.0-7.1.el7.noarch                                                                             2/2
  Verifying  : redhat-virtualization-host-image-update-4.1-20170727.1.el7_4.noarch                                                                                1/2
  Verifying  : redhat-virtualization-host-image-update-placeholder-4.0-7.1.el7.noarch                                                                             2/2

Installed:
  redhat-virtualization-host-image-update.noarch 0:4.1-20170727.1.el7_4                                                                                              

Replaced:
  redhat-virtualization-host-image-update-placeholder.noarch 0:4.0-7.1.el7                                                                                            

Complete!

2. After step 4, missing boot entry of rhvh-4.1-20170727.1



Expected results:
1. After step 3, there should be no error info during upgrade
2. After step 4, there should be boot entry of rhvh-4.1-20170727.1 after upgrade


Additional info:
1. Also encountered this issue when upgrade from rhvh-4.1-20170616.0 to rhvh-4.1-20170727.1
2. Also encountered this issue when upgrade via "# yum install redhat-virtualization-host-image-update-4.1-20170727.1.el7_4.noarch.rpm"
3. No such issue in rhvh-4.1-20170721.0, so this is regression issue
Comment 1 Ryan Barry 2017-07-28 00:28:25 EDT
I wish I had a good explanation for this, but I don't -- nothing in the imgbased code touches this.

Lazy umounting resolves. I'd expect that "lsof /etc" shows udev holding files after the upgrade fails, which has not occurred before.
Comment 2 Red Hat Bugzilla Rules Engine 2017-07-28 00:28:33 EDT
This bug report has Keywords: Regression or TestBlocker.
Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.
Comment 3 Huijuan Zhao 2017-07-30 22:56:09 EDT
Test version:
From: redhat-virtualization-host-4.0-20170307.1
To:   redhat-virtualization-host-4.1-20170728.0
      imgbased-0.9.36-0.1.el7ev.noarch


Test steps:
Same as comment 0


Test results:
1. After step 3, there is no error info during upgrade
2. After step 4, there is boot entry of rhvh-4.1-20170727.1 after upgrade


So this bug is fixed in imgbased-0.9.36-0.1.el7ev.noarch, change the status to VERIFIED.

Note You need to log in before you can comment on or make changes to this bug.