Bug 1514625

Summary: Failed to upgrade ovirt-node-ng 4.1.6 to 4.1.7 (ovirt-node-ng-image-update-4.1.7-1)
Product: [oVirt] ovirt-node Reporter: Mike Goodwin <mike>
Component: Installation & UpdateAssignee: Ryan Barry <rbarry>
Status: CLOSED CURRENTRELEASE QA Contact: Huijuan Zhao <huzhao>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 4.1CC: bugs, cshao, dguo, huzhao, jiawu, lveyde, mike, qiyuan, rbarry, sbonazzo, vgshved, weiwang, yaniwang, ycui, yturgema, yzhao
Target Milestone: ovirt-4.1.7Keywords: TestBlocker, Upstream
Target Release: ---Flags: rule-engine: ovirt-4.1+
rule-engine: blocker+
huzhao: testing_ack+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1510858 Environment:
Last Closed: 2017-11-20 17:02:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Node RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Mike Goodwin 2017-11-17 23:05:08 UTC 
+++ This bug was initially created as a clone of Bug #1510858 +++

Description of problem:
Failed to upgrade ovirt-node from 4.1.6 to 4.1.7 the ovirt-node-ng 4.1.7 (ovirt-node-ng-installer-ovirt-4.1-2017110715.iso)

2017-11-17 22:53:22,436 [DEBUG] (MainThread) Returned:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/tmp/tmp.Nxdp3FKvmr/usr/lib/python2.7/site-packages/imgbased/__main__.py", line 53, in <module>
    CliApplication()
  File "/tmp/tmp.Nxdp3FKvmr/usr/lib/python2.7/site-packages/imgbased/__init__.py", line 82, in CliApplication
    app.hooks.emit("post-arg-parse", args)
  File "/tmp/tmp.Nxdp3FKvmr/usr/lib/python2.7/site-packages/imgbased/hooks.py", line 120, in emit
    cb(self.context, *args)
  File "/tmp/tmp.Nxdp3FKvmr/usr/lib/python2.7/site-packages/imgbased/plugins/update.py", line 56, in post_argparse
    base_lv, _ = LiveimgExtractor(app.imgbase).extract(args.FILENAME)
  File "/tmp/tmp.Nxdp3FKvmr/usr/lib/python2.7/site-packages/imgbased/plugins/update.py", line 118, in extract
    "%s" % size, nvr)
  File "/tmp/tmp.Nxdp3FKvmr/usr/lib/python2.7/site-packages/imgbased/plugins/update.py", line 99, in add_base_with_tree
    new_layer_lv = self.imgbase.add_layer(new_base)
  File "/tmp/tmp.Nxdp3FKvmr/usr/lib/python2.7/site-packages/imgbased/imgbase.py", line 191, in add_layer
    self.hooks.emit("new-layer-added", prev_lv, new_lv)
  File "/tmp/tmp.Nxdp3FKvmr/usr/lib/python2.7/site-packages/imgbased/hooks.py", line 120, in emit
    cb(self.context, *args)
  File "/tmp/tmp.Nxdp3FKvmr/usr/lib/python2.7/site-packages/imgbased/plugins/osupdater.py", line 123, in on_new_layer
    raise ConfigMigrationError()
imgbased.plugins.osupdater.ConfigMigrationError

Version-Release number of selected component (if applicable):
ovirt-node-ng-image-update-4.1.7-1.el7.centos.noarch


How reproducible:
100%

Steps to Reproduce:
1. Start with ovirt-node version 4.1.6
2. `yum upgrade` to bring ovirt-node-ng-image-update up-to-date
3. See below

Actual results:
Dependencies Resolved

==========================================================================================================================================================
 Package                                         Arch                        Version                                 Repository                      Size
==========================================================================================================================================================
Reinstalling:
 ovirt-node-ng-image-update                      noarch                      4.1.7-1.el7.centos                      ovirt-4.1                      585 M

Transaction Summary
==========================================================================================================================================================
Reinstall  1 Package

Total download size: 585 M
Installed size: 585 M
Is this ok [y/d/N]: y
Downloading packages:
ovirt-node-ng-image-update-4.1.7-1.el7.centos.noarch.rpm                                                                           | 585 MB  00:01:30     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : ovirt-node-ng-image-update-4.1.7-1.el7.centos.noarch                                                                                   1/1 
warning: %post(ovirt-node-ng-image-update-4.1.7-1.el7.centos.noarch) scriptlet failed, exit status 1
Non-fatal POSTIN scriptlet failure in rpm package ovirt-node-ng-image-update-4.1.7-1.el7.centos.noarch
  Verifying  : ovirt-node-ng-image-update-4.1.7-1.el7.centos.noarch                                                                                   1/1 

Installed:
  ovirt-node-ng-image-update.noarch 0:4.1.7-1.el7.centos                                                                                                  

Complete!



Then, nodectl info shows the following:

[root@ovn-vmh-r810-1 yum.repos.d]# nodectl info
layers:
  ovirt-node-ng-4.1.7-0.20171107.0:
    ovirt-node-ng-4.1.7-0.20171107.0+1
  ovirt-node-ng-4.1.6-0.20170919.0:
    ovirt-node-ng-4.1.6-0.20170919.0+1
bootloader:
  default: ovirt-node-ng-4.1.6-0.20170919.0+1
  entries:
    ovirt-node-ng-4.1.6-0.20170919.0+1:
      index: 0
      title: ovirt-node-ng-4.1.6-0.20170919.0
      kernel: /boot/ovirt-node-ng-4.1.6-0.20170919.0+1/vmlinuz-3.10.0-693.2.2.el7.x86_64
      args: "ro crashkernel=auto rd.lvm.lv=onn/ovirt-node-ng-4.1.6-0.20170919.0+1 rd.lvm.lv=onn/swap rhgb quiet LANG=en_US.UTF-8 img.bootid=ovirt-node-ng-4.1.6-0.20170919.0+1"
      initrd: /boot/ovirt-node-ng-4.1.6-0.20170919.0+1/initramfs-3.10.0-693.2.2.el7.x86_64.img
      root: /dev/onn/ovirt-node-ng-4.1.6-0.20170919.0+1
current_layer: ovirt-node-ng-4.1.6-0.20170919.0+1



Expected results:
1. Yum upgrade completes without errors and 4.1.7 becomes the active layer on next reboot.

Since this has supposedly been fixed in the ISO... do the packages just need to be rebuilt and deployed? Are my expectations off that if it's fixed in the iso the packages should be updated to match?
Comment 1 Red Hat Bugzilla Rules Engine 2017-11-17 23:05:33 UTC 
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.
Comment 2 Red Hat Bugzilla Rules Engine 2017-11-17 23:05:33 UTC 
This bug report has Keywords: Regression or TestBlocker.
Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.
Comment 3 Ryan Barry 2017-11-18 01:36:39 UTC 
Sandro/Lev, was something missed in publishing here?

releng-tools was updated to point to the correct Jenkins build:

https://gerrit.ovirt.org/#/c/83811/

The manifest there shows that it includes imgbased-0.9.50

However, the RPM I pull if I 'yum -y --downloadonly --downloaddir=. update ovirt-node-ng-image-update' includes imgbased-0.9.49, and it's definitely an earlier build:

><fs> sh 'cat /usr/share/imgbase/build/meta/nvr' 
ovirt-node-ng-4.1.7-0.20171107.0   

Can we re-run the publisher?
Comment 4 Huijuan Zhao 2017-11-20 07:06:21 UTC 
QE can reproduce this issue.

Test version:
From: ovirt-node-ng-4.1.6-0.20170919.0
To:   ovirt-node-ng-4.1.7-0.20171107.0

Test steps:
1. Install ovirt-node-ng-4.1.6-0.20170919.0
2. Download 4.1.7 update rpm, and upgrade host:
# yum install ovirt-node-ng-image-update-4.1.7-1.el7.centos.noarch.rpm
3. Focus the upgrade process and check /tmp/imgbased.log


Actual results:
In step 3, there is failure info during upgrade, almost same as comment 0.

# yum install ovirt-node-ng-image-update-4.1.7-1.el7.centos.noarch.rpm 
Loaded plugins: fastestmirror, imgbased-persist
Examining ovirt-node-ng-image-update-4.1.7-1.el7.centos.noarch.rpm: ovirt-node-ng-image-update-4.1.7-1.el7.centos.noarch
Marking ovirt-node-ng-image-update-4.1.7-1.el7.centos.noarch.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package ovirt-node-ng-image-update.noarch 0:4.1.7-1.el7.centos will be obsoleting
---> Package ovirt-node-ng-image-update-placeholder.noarch 0:4.1.6-1.el7.centos will be obsoleted
--> Finished Dependency Resolution

Dependencies Resolved

======================================================================================================================================================================
 Package                                 Arch                Version                         Repository                                                          Size
======================================================================================================================================================================
Installing:
 ovirt-node-ng-image-update              noarch              4.1.7-1.el7.centos              /ovirt-node-ng-image-update-4.1.7-1.el7.centos.noarch              585 M
     replacing  ovirt-node-ng-image-update-placeholder.noarch 4.1.6-1.el7.centos

Transaction Summary
======================================================================================================================================================================
Install  1 Package

Total size: 585 M
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : ovirt-node-ng-image-update-4.1.7-1.el7.centos.noarch                                                                                               1/2 
warning: %post(ovirt-node-ng-image-update-4.1.7-1.el7.centos.noarch) scriptlet failed, exit status 1
Non-fatal POSTIN scriptlet failure in rpm package ovirt-node-ng-image-update-4.1.7-1.el7.centos.noarch
  Erasing    : ovirt-node-ng-image-update-placeholder-4.1.6-1.el7.centos.noarch                                                                                   2/2 
  Verifying  : ovirt-node-ng-image-update-4.1.7-1.el7.centos.noarch                                                                                               1/2 
  Verifying  : ovirt-node-ng-image-update-placeholder-4.1.6-1.el7.centos.noarch                                                                                   2/2 

Installed:
  ovirt-node-ng-image-update.noarch 0:4.1.7-1.el7.centos                                                                                                              

Replaced:
  ovirt-node-ng-image-update-placeholder.noarch 0:4.1.6-1.el7.centos                                                                                                  

Complete!


Expected results:
In step3, should upgrade successful.
Comment 5 Lev Veyde 2017-11-20 16:52:52 UTC 
(In reply to Ryan Barry from comment #3)
> Sandro/Lev, was something missed in publishing here?
> 
> releng-tools was updated to point to the correct Jenkins build:
> 
> https://gerrit.ovirt.org/#/c/83811/
> 
> The manifest there shows that it includes imgbased-0.9.50
> 
> However, the RPM I pull if I 'yum -y --downloadonly --downloaddir=. update
> ovirt-node-ng-image-update' includes imgbased-0.9.49, and it's definitely an
> earlier build:
> 
> ><fs> sh 'cat /usr/share/imgbase/build/meta/nvr' 
> ovirt-node-ng-4.1.7-0.20171107.0   
> 
> Can we re-run the publisher?

Looks like an issue caused by a bug in a repoman.
Manually re-composed the repo with the correct package, so the issue should be resolved now.
Comment 6 Ryan Barry 2017-11-20 17:02:15 UTC 
This was re-published, and upgrades from yum now also succeed.
Comment 7 Mike Goodwin 2017-11-20 18:09:21 UTC 
Leaves me wondering whether this will continue to fail for the next upgrade? While I'm sure this exact issue is resolved, is the parent issue of it not updating the packages in the first place also resolved?
Comment 8 Ryan Barry 2017-11-20 19:18:29 UTC 
Hi Mike -

This will not continue to fail. There was no parent issue.

The tracebacks here look similar, but are not identical. Since imgbased threads updates, many tracebacks look similar without a full log, unfortunately.

The root cause in this case is that a patch to migrate Satellite files from /usr/share/rhn threw an exception upstream (since /usr/share/rhn did not exist). New packages were built the day after 4.1.7 released, but a previously unknown problem with the upstream publisher prevented the new packages from making into the repos.

We're good going forward.
Comment 9 Mike Goodwin 2017-11-28 23:10:07 UTC 
You guys should have bumped the spec so you wouldn't have to `yum reinstall ovirt-node-ng-image-update`, since anyone like me who did and it failed already have 4.1.7-1 and it doesn't see the re-publish as an upgrade. 

It did however upgrade the node properly this time, albeit there were errors like before in the postin scriplet, but I assume that is due to something the previous 4.1.7 upgrade did partially. Regardless, it succeeded. Thanks