+++ This bug was initially created as a clone of Bug #1510858 +++ Description of problem: Failed to upgrade ovirt-node from 4.1.6 to 4.1.7 the ovirt-node-ng 4.1.7 (ovirt-node-ng-installer-ovirt-4.1-2017110715.iso) 2017-11-17 22:53:22,436 [DEBUG] (MainThread) Returned: Traceback (most recent call last): File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/tmp/tmp.Nxdp3FKvmr/usr/lib/python2.7/site-packages/imgbased/__main__.py", line 53, in <module> CliApplication() File "/tmp/tmp.Nxdp3FKvmr/usr/lib/python2.7/site-packages/imgbased/__init__.py", line 82, in CliApplication app.hooks.emit("post-arg-parse", args) File "/tmp/tmp.Nxdp3FKvmr/usr/lib/python2.7/site-packages/imgbased/hooks.py", line 120, in emit cb(self.context, *args) File "/tmp/tmp.Nxdp3FKvmr/usr/lib/python2.7/site-packages/imgbased/plugins/update.py", line 56, in post_argparse base_lv, _ = LiveimgExtractor(app.imgbase).extract(args.FILENAME) File "/tmp/tmp.Nxdp3FKvmr/usr/lib/python2.7/site-packages/imgbased/plugins/update.py", line 118, in extract "%s" % size, nvr) File "/tmp/tmp.Nxdp3FKvmr/usr/lib/python2.7/site-packages/imgbased/plugins/update.py", line 99, in add_base_with_tree new_layer_lv = self.imgbase.add_layer(new_base) File "/tmp/tmp.Nxdp3FKvmr/usr/lib/python2.7/site-packages/imgbased/imgbase.py", line 191, in add_layer self.hooks.emit("new-layer-added", prev_lv, new_lv) File "/tmp/tmp.Nxdp3FKvmr/usr/lib/python2.7/site-packages/imgbased/hooks.py", line 120, in emit cb(self.context, *args) File "/tmp/tmp.Nxdp3FKvmr/usr/lib/python2.7/site-packages/imgbased/plugins/osupdater.py", line 123, in on_new_layer raise ConfigMigrationError() imgbased.plugins.osupdater.ConfigMigrationError Version-Release number of selected component (if applicable): ovirt-node-ng-image-update-4.1.7-1.el7.centos.noarch How reproducible: 100% Steps to Reproduce: 1. Start with ovirt-node version 4.1.6 2. `yum upgrade` to bring ovirt-node-ng-image-update up-to-date 3. See below Actual results: Dependencies Resolved ========================================================================================================================================================== Package Arch Version Repository Size ========================================================================================================================================================== Reinstalling: ovirt-node-ng-image-update noarch 4.1.7-1.el7.centos ovirt-4.1 585 M Transaction Summary ========================================================================================================================================================== Reinstall 1 Package Total download size: 585 M Installed size: 585 M Is this ok [y/d/N]: y Downloading packages: ovirt-node-ng-image-update-4.1.7-1.el7.centos.noarch.rpm | 585 MB 00:01:30 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : ovirt-node-ng-image-update-4.1.7-1.el7.centos.noarch 1/1 warning: %post(ovirt-node-ng-image-update-4.1.7-1.el7.centos.noarch) scriptlet failed, exit status 1 Non-fatal POSTIN scriptlet failure in rpm package ovirt-node-ng-image-update-4.1.7-1.el7.centos.noarch Verifying : ovirt-node-ng-image-update-4.1.7-1.el7.centos.noarch 1/1 Installed: ovirt-node-ng-image-update.noarch 0:4.1.7-1.el7.centos Complete! Then, nodectl info shows the following: [root@ovn-vmh-r810-1 yum.repos.d]# nodectl info layers: ovirt-node-ng-4.1.7-0.20171107.0: ovirt-node-ng-4.1.7-0.20171107.0+1 ovirt-node-ng-4.1.6-0.20170919.0: ovirt-node-ng-4.1.6-0.20170919.0+1 bootloader: default: ovirt-node-ng-4.1.6-0.20170919.0+1 entries: ovirt-node-ng-4.1.6-0.20170919.0+1: index: 0 title: ovirt-node-ng-4.1.6-0.20170919.0 kernel: /boot/ovirt-node-ng-4.1.6-0.20170919.0+1/vmlinuz-3.10.0-693.2.2.el7.x86_64 args: "ro crashkernel=auto rd.lvm.lv=onn/ovirt-node-ng-4.1.6-0.20170919.0+1 rd.lvm.lv=onn/swap rhgb quiet LANG=en_US.UTF-8 img.bootid=ovirt-node-ng-4.1.6-0.20170919.0+1" initrd: /boot/ovirt-node-ng-4.1.6-0.20170919.0+1/initramfs-3.10.0-693.2.2.el7.x86_64.img root: /dev/onn/ovirt-node-ng-4.1.6-0.20170919.0+1 current_layer: ovirt-node-ng-4.1.6-0.20170919.0+1 Expected results: 1. Yum upgrade completes without errors and 4.1.7 becomes the active layer on next reboot. Since this has supposedly been fixed in the ISO... do the packages just need to be rebuilt and deployed? Are my expectations off that if it's fixed in the iso the packages should be updated to match?
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.
This bug report has Keywords: Regression or TestBlocker. Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.
Sandro/Lev, was something missed in publishing here? releng-tools was updated to point to the correct Jenkins build: https://gerrit.ovirt.org/#/c/83811/ The manifest there shows that it includes imgbased-0.9.50 However, the RPM I pull if I 'yum -y --downloadonly --downloaddir=. update ovirt-node-ng-image-update' includes imgbased-0.9.49, and it's definitely an earlier build: ><fs> sh 'cat /usr/share/imgbase/build/meta/nvr' ovirt-node-ng-4.1.7-0.20171107.0 Can we re-run the publisher?
QE can reproduce this issue. Test version: From: ovirt-node-ng-4.1.6-0.20170919.0 To: ovirt-node-ng-4.1.7-0.20171107.0 Test steps: 1. Install ovirt-node-ng-4.1.6-0.20170919.0 2. Download 4.1.7 update rpm, and upgrade host: # yum install ovirt-node-ng-image-update-4.1.7-1.el7.centos.noarch.rpm 3. Focus the upgrade process and check /tmp/imgbased.log Actual results: In step 3, there is failure info during upgrade, almost same as comment 0. # yum install ovirt-node-ng-image-update-4.1.7-1.el7.centos.noarch.rpm Loaded plugins: fastestmirror, imgbased-persist Examining ovirt-node-ng-image-update-4.1.7-1.el7.centos.noarch.rpm: ovirt-node-ng-image-update-4.1.7-1.el7.centos.noarch Marking ovirt-node-ng-image-update-4.1.7-1.el7.centos.noarch.rpm to be installed Resolving Dependencies --> Running transaction check ---> Package ovirt-node-ng-image-update.noarch 0:4.1.7-1.el7.centos will be obsoleting ---> Package ovirt-node-ng-image-update-placeholder.noarch 0:4.1.6-1.el7.centos will be obsoleted --> Finished Dependency Resolution Dependencies Resolved ====================================================================================================================================================================== Package Arch Version Repository Size ====================================================================================================================================================================== Installing: ovirt-node-ng-image-update noarch 4.1.7-1.el7.centos /ovirt-node-ng-image-update-4.1.7-1.el7.centos.noarch 585 M replacing ovirt-node-ng-image-update-placeholder.noarch 4.1.6-1.el7.centos Transaction Summary ====================================================================================================================================================================== Install 1 Package Total size: 585 M Is this ok [y/d/N]: y Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : ovirt-node-ng-image-update-4.1.7-1.el7.centos.noarch 1/2 warning: %post(ovirt-node-ng-image-update-4.1.7-1.el7.centos.noarch) scriptlet failed, exit status 1 Non-fatal POSTIN scriptlet failure in rpm package ovirt-node-ng-image-update-4.1.7-1.el7.centos.noarch Erasing : ovirt-node-ng-image-update-placeholder-4.1.6-1.el7.centos.noarch 2/2 Verifying : ovirt-node-ng-image-update-4.1.7-1.el7.centos.noarch 1/2 Verifying : ovirt-node-ng-image-update-placeholder-4.1.6-1.el7.centos.noarch 2/2 Installed: ovirt-node-ng-image-update.noarch 0:4.1.7-1.el7.centos Replaced: ovirt-node-ng-image-update-placeholder.noarch 0:4.1.6-1.el7.centos Complete! Expected results: In step3, should upgrade successful.
(In reply to Ryan Barry from comment #3) > Sandro/Lev, was something missed in publishing here? > > releng-tools was updated to point to the correct Jenkins build: > > https://gerrit.ovirt.org/#/c/83811/ > > The manifest there shows that it includes imgbased-0.9.50 > > However, the RPM I pull if I 'yum -y --downloadonly --downloaddir=. update > ovirt-node-ng-image-update' includes imgbased-0.9.49, and it's definitely an > earlier build: > > ><fs> sh 'cat /usr/share/imgbase/build/meta/nvr' > ovirt-node-ng-4.1.7-0.20171107.0 > > Can we re-run the publisher? Looks like an issue caused by a bug in a repoman. Manually re-composed the repo with the correct package, so the issue should be resolved now.
This was re-published, and upgrades from yum now also succeed.
Leaves me wondering whether this will continue to fail for the next upgrade? While I'm sure this exact issue is resolved, is the parent issue of it not updating the packages in the first place also resolved?
Hi Mike - This will not continue to fail. There was no parent issue. The tracebacks here look similar, but are not identical. Since imgbased threads updates, many tracebacks look similar without a full log, unfortunately. The root cause in this case is that a patch to migrate Satellite files from /usr/share/rhn threw an exception upstream (since /usr/share/rhn did not exist). New packages were built the day after 4.1.7 released, but a previously unknown problem with the upstream publisher prevented the new packages from making into the repos. We're good going forward.
You guys should have bumped the spec so you wouldn't have to `yum reinstall ovirt-node-ng-image-update`, since anyone like me who did and it failed already have 4.1.7-1 and it doesn't see the re-publish as an upgrade. It did however upgrade the node properly this time, albeit there were errors like before in the postin scriplet, but I assume that is due to something the previous 4.1.7 upgrade did partially. Regardless, it succeeded. Thanks