Bug 1427468 - Upgrade from wrapper to wrapper failed (ConfigMigrationError)
Summary: Upgrade from wrapper to wrapper failed (ConfigMigrationError)
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-node
Classification: oVirt
Component: Installation & Update
Version: 4.1
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ovirt-4.1.1
: 4.1
Assignee: Ryan Barry
QA Contact: Huijuan Zhao
URL:
Whiteboard:
: 1427478 (view as bug list)
Depends On:
Blocks: 1419535
TreeView+ depends on / blocked
 
Reported: 2017-02-28 09:54 UTC by Huijuan Zhao
Modified: 2019-09-29 08:20 UTC (History)
15 users (show)

Fixed In Version: imgbased-0.9.18-0.1.el7ev
Clone Of:
Environment:
Last Closed: 2017-04-21 09:38:25 UTC
oVirt Team: Node
Embargoed:
rule-engine: ovirt-4.1+
rule-engine: blocker+
rule-engine: planning_ack+
sbonazzo: devel_ack+
ycui: testing_ack+


Attachments (Terms of Use)
/tmp/imgbased.log (89.56 KB, text/plain)
2017-02-28 09:54 UTC, Huijuan Zhao
no flags Details
comment 7 : sosreport and log (12.01 MB, application/x-gzip)
2017-03-14 05:06 UTC, Huijuan Zhao
no flags Details


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 73212 0 ovirt-4.1 ABANDONED Revert "osupdater: migrate to NIST partitioning" 2020-12-15 10:34:15 UTC
oVirt gerrit 73278 0 master MERGED utils: add _ismounted to MountPoint 2020-12-15 10:34:15 UTC
oVirt gerrit 73539 0 ovirt-4.1 MERGED utils: add _ismounted to MountPoint 2020-12-15 10:34:15 UTC
oVirt gerrit 74132 0 master MERGED osupdater: exclude critical files from remediation 2020-12-15 10:34:15 UTC
oVirt gerrit 74141 0 ovirt-4.1 MERGED osupdater: exclude critical files from remediation 2020-12-15 10:34:15 UTC

Description Huijuan Zhao 2017-02-28 09:54:58 UTC
Created attachment 1258310 [details]
/tmp/imgbased.log

Description of problem:
Upgrade from wrapper to wrapper failed: 
1. There is error info during "yum update"
2. There is 3 bases/layers, which should be 2 bases/layers
3. Miss boot entry of latest build

Please see /tmp/imgbased.log, you'd see that rsync is running out of space which raises a ConfigMigrationError.


Test version:
Build1:
redhat-virtualization-host-4.0-20160919.0
Build2:
redhat-virtualization-host-4.0-20161116.0
Build3:
redhat-virtualization-host-4.1-20170222.0

# imgbase layout
rhvh-4.0-0.20160919.0
 +- rhvh-4.0-0.20160919.0+1
rhvh-4.0-0.20161116.0
 +- rhvh-4.0-0.20161116.0+1
rhvh-4.1-0.20170223.0
 +- rhvh-4.1-0.20170223.0+1

Test steps:
1. Install build1 redhat-virtualization-host-4.0-20160919.0
2. Reboot and login build1, download redhat-virtualization-host-image-update-4.0-20161116.1.el7_3.noarch.rpm and update:
# yum install redhat-virtualization-host-image-update-4.0-20161116.1.el7_3.noarch.rpm
3. Reboot and login build2 rhvh-4.0-0.20161116.0, setup local repos, update to build3 redhat-virtualization-host-4.1-20170222.0:
# yum update
4. Reboot and focus on boot entry

Actual results:
1. In step3, there is error info during update "#yum update". After update,there are 3 bases.

# yum update
Loaded plugins: imgbased-warning, product-id, search-disabled-repos, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Warning: yum operations are not persisted across upgrades!
Repo rhel-7-server-rhvh-4-source-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877.pem
Repo rhel-7-server-rhvh-4-source-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877-key.pem
Repo rhel-7-server-rhvh-4-beta-debug-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877.pem
Repo rhel-7-server-rhvh-4-beta-debug-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877-key.pem
Repo rhel-7-server-rhvh-4-debug-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877.pem
Repo rhel-7-server-rhvh-4-debug-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877-key.pem
Repo rhel-7-server-rhvh-4-beta-source-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877.pem
Repo rhel-7-server-rhvh-4-beta-source-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877-key.pem
Repo rhel-7-server-rhvh-4-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877.pem
Repo rhel-7-server-rhvh-4-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877-key.pem
Repo rhel-7-server-rhvh-4-beta-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877.pem
Repo rhel-7-server-rhvh-4-beta-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877-key.pem
rhel-7-server-rhvh-4-beta-debug-rpms                                                                                                           | 2.9 kB  00:00:00     
rhel-7-server-rhvh-4-beta-rpms                                                                                                                 | 2.9 kB  00:00:00     
rhel-7-server-rhvh-4-beta-source-rpms                                                                                                          | 2.9 kB  00:00:00     
rhel-7-server-rhvh-4-debug-rpms                                                                                                                | 2.9 kB  00:00:00     
rhel-7-server-rhvh-4-rpms                                                                                                                      | 2.9 kB  00:00:00     
rhel-7-server-rhvh-4-source-rpms                                                                                                               | 2.9 kB  00:00:00     
(1/6): rhel-7-server-rhvh-4-beta-debug-rpms/primary_db                                                                                         | 1.7 kB  00:00:00     
(2/6): rhel-7-server-rhvh-4-source-rpms/primary_db                                                                                             | 4.1 kB  00:00:00     
(3/6): rhel-7-server-rhvh-4-beta-rpms/primary_db                                                                                               | 9.3 kB  00:00:00     
(4/6): rhel-7-server-rhvh-4-debug-rpms/primary_db                                                                                              | 1.7 kB  00:00:00     
(5/6): rhel-7-server-rhvh-4-rpms/primary_db                                                                                                    | 9.3 kB  00:00:00     
(6/6): rhel-7-server-rhvh-4-beta-source-rpms/primary_db                                                                                        | 4.1 kB  00:00:00     
Resolving Dependencies
--> Running transaction check
---> Package redhat-virtualization-host-image-update.noarch 0:4.1-20170222.0.el7_3 will be obsoleting
---> Package redhat-virtualization-host-image-update-placeholder.noarch 0:4.0-5.2.el7 will be obsoleted
--> Finished Dependency Resolution

Dependencies Resolved

======================================================================================================================================================================
 Package                                                Arch                  Version                             Repository                                     Size
======================================================================================================================================================================
Installing:
 redhat-virtualization-host-image-update                noarch                4.1-20170222.0.el7_3                rhel-7-server-rhvh-4-beta-rpms                548 M
     replacing  redhat-virtualization-host-image-update-placeholder.noarch 4.0-5.2.el7

Transaction Summary
======================================================================================================================================================================
Install  1 Package

Total download size: 548 M
Is this ok [y/d/N]: y
Downloading packages:
redhat-virtualization-host-image-update-latest.rpm                                                                                             | 548 MB  00:00:07     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : redhat-virtualization-host-image-update-4.1-20170222.0.el7_3.noarch                                                                                1/2 
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
lsof: no pwd entry for UID 992
cp: warning: source file ‘/var/cache/yum/x86_64/7RedHatVirtualizationHost/rhel-7-server-rhvh-4-beta-rpms/packages/redhat-virtualization-host-image-update-latest.rpm’ specified more than once
  Erasing    : redhat-virtualization-host-image-update-placeholder-4.0-5.2.el7.noarch                                                                             2/2 
  Verifying  : redhat-virtualization-host-image-update-4.1-20170222.0.el7_3.noarch                                                                                1/2 
  Verifying  : redhat-virtualization-host-image-update-placeholder-4.0-5.2.el7.noarch                                                                             2/2 

Installed:
  redhat-virtualization-host-image-update.noarch 0:4.1-20170222.0.el7_3                                                                                               

Replaced:
  redhat-virtualization-host-image-update-placeholder.noarch 0:4.0-5.2.el7                                                                                            

Complete!


[root@dhcp-10-16 ~]# imgbase w
[INFO] You are on rhvh-4.0-0.20161116.0+1

[root@dhcp-10-16 ~]# imgbase layout
rhvh-4.0-0.20160919.0
 +- rhvh-4.0-0.20160919.0+1
rhvh-4.0-0.20161116.0
 +- rhvh-4.0-0.20161116.0+1
rhvh-4.1-0.20170223.0
 +- rhvh-4.1-0.20170223.0+1

2. In step4, there is not boot entry of build3 rhvh-4.1-0.20170223.0, only 2 boot entry of rhvh-4.0-0.20160919.0 and rhvh-4.0-0.20161116.0.
And login rhvh-4.0-0.20161116.0, encounter Bug 1426151.

Expected results:
1. In step3, there should be no error info during update. After upgrade, there should be only 2 latest bases.
2. In step4, there should be boot entry of rhvh-4.1-0.20170223.0 and rhvh-4.0-0.20161116.0


Additional info:

Comment 1 Ryan Barry 2017-02-28 14:50:23 UTC
*** Bug 1427478 has been marked as a duplicate of this bug. ***

Comment 3 Red Hat Bugzilla Rules Engine 2017-03-10 08:24:26 UTC
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 5 Ryan Barry 2017-03-13 20:18:28 UTC
I just tried updating:

4.0-20160916.0
4.0-20161116.1
4.1-20170308.1

Still can't reproduce...

Another reproducer would be great, if possible.

Comment 6 cshao 2017-03-14 02:18:23 UTC
huzhao,

Could you help to reply #c5?
Thanks.

Comment 7 Huijuan Zhao 2017-03-14 05:04:23 UTC
(In reply to Ryan Barry from comment #5)
> I just tried updating:
> 
> 4.0-20160916.0
> 4.0-20161116.1
> 4.1-20170308.1
> 
> Still can't reproduce...
> 
> Another reproducer would be great, if possible.

Ryan, I reproduced again with another machine.

4.0-20160919.0
4.0-20161116.1
4.1-20170308.1

I modified /etc/ssh/ssh_host_ed25519_key to /etc/ssh/ssh_host_ed25519_key_backup, then can ssh successful.

Please refer to attachment for log.
And I will send ENV info via email.

Comment 8 Huijuan Zhao 2017-03-14 05:06:02 UTC
Created attachment 1262772 [details]
comment 7 : sosreport and log

Comment 9 Qin Yuan 2017-03-14 06:37:47 UTC
I tried to reproduce this issue on my local testing machine, and it can be reproduced. The three builds are the same as comment #7.

After upgrade to 4.1-20170308.1, I checked sshd.service and /etc/ssh:

# systemctl status sshd
● sshd.service - OpenSSH server daemon
   Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2017-03-14 14:18:16 CST; 1min 1s ago
     Docs: man:sshd(8)
           man:sshd_config(5)
  Process: 21658 ExecStart=/usr/sbin/sshd $OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 21660 (sshd)
   CGroup: /system.slice/sshd.service
           └─21660 /usr/sbin/sshd

Mar 14 14:18:16 dhcp-9-139.nay.redhat.com sshd[21658]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 14:18:16 dhcp-9-139.nay.redhat.com sshd[21658]: Permissions 0640 for '/etc/ssh/ssh_host_ed25519_key' are too open.
Mar 14 14:18:16 dhcp-9-139.nay.redhat.com sshd[21658]: It is required that your private key files are NOT accessible by others.
Mar 14 14:18:16 dhcp-9-139.nay.redhat.com sshd[21658]: This private key will be ignored.
Mar 14 14:18:16 dhcp-9-139.nay.redhat.com sshd[21658]: bad permissions: ignore key: /etc/ssh/ssh_host_ed25519_key
Mar 14 14:18:16 dhcp-9-139.nay.redhat.com sshd[21658]: Could not load host key: /etc/ssh/ssh_host_ed25519_key
Mar 14 14:18:16 dhcp-9-139.nay.redhat.com systemd[1]: PID file /var/run/sshd.pid not readable (yet?) after start.
Mar 14 14:18:16 dhcp-9-139.nay.redhat.com sshd[21660]: Server listening on 0.0.0.0 port 22.
Mar 14 14:18:16 dhcp-9-139.nay.redhat.com systemd[1]: Started OpenSSH server daemon.
Mar 14 14:18:16 dhcp-9-139.nay.redhat.com sshd[21660]: Server listening on :: port 22.

# ls -al /etc/ssh
drwxr-xr-x.   2 root root     225 Mar 14 14:06 .
drwxr-xr-x. 118 root root    8192 Mar 14 14:16 ..
-rw-r--r--.   1 root root  242153 Dec 21 00:27 moduli
-rw-r--r--.   1 root root    2208 Dec 21 00:27 ssh_config
-rw-------.   1 root root    4361 Dec 21 00:27 sshd_config
-rw-r-----.   1 root input    227 Mar 14 05:17 ssh_host_ecdsa_key
-rw-r--r--.   1 root root     162 Mar 14 05:17 ssh_host_ecdsa_key.pub
-rw-r-----.   1 root input    387 Mar 14 05:17 ssh_host_ed25519_key
-rw-r--r--.   1 root root      82 Mar 14 05:17 ssh_host_ed25519_key.pub
-rw-r-----.   1 root input   1675 Mar 14 05:17 ssh_host_rsa_key
-rw-r--r--.   1 root root     382 Mar 14 05:17 ssh_host_rsa_key.pub


The group of ssh_host_ecdsa_key, ssh_host_ed25519_key, and ssh_host_rsa_key is input. This is incorrect, it should be ssh_keys. 

I changed the group to ssh_keys, using cmd:
# chown root:ssh_keys ssh_host_ecdsa_key
# chown root:ssh_keys ssh_host_ed25519_key
# chown root:ssh_keys ssh_host_rsa_key

and restart sshd using cmd:
# systemctl restart sshd

recheck sshd.service:
# systemctl status sshd
● sshd.service - OpenSSH server daemon
   Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2017-03-14 14:22:09 CST; 17s ago
     Docs: man:sshd(8)
           man:sshd_config(5)
  Process: 21698 ExecStart=/usr/sbin/sshd $OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 21700 (sshd)
   CGroup: /system.slice/sshd.service
           └─21700 /usr/sbin/sshd

Mar 14 14:22:09 dhcp-9-139.nay.redhat.com systemd[1]: Starting OpenSSH server daemon...
Mar 14 14:22:09 dhcp-9-139.nay.redhat.com systemd[1]: PID file /var/run/sshd.pid not readable (yet?) after start.
Mar 14 14:22:09 dhcp-9-139.nay.redhat.com sshd[21700]: Server listening on 0.0.0.0 port 22.
Mar 14 14:22:09 dhcp-9-139.nay.redhat.com sshd[21700]: Server listening on :: port 22.
Mar 14 14:22:09 dhcp-9-139.nay.redhat.com systemd[1]: Started OpenSSH server daemon.

Finally, this issue has been resloved by this workaroud, sshd returns to normal status.

Comment 10 Huijuan Zhao 2017-03-16 05:32:39 UTC
Test version:
Build1:
redhat-virtualization-host-4.0-20160919.0
Build2:
redhat-virtualization-host-4.0-20161116.0
Build3:
redhat-virtualization-host-4.1-20170314.0
imgbased-0.9.18-0.1.el7ev.noarch


Test steps:
1. Install build1 redhat-virtualization-host-4.0-20160919.0
2. Reboot and login build1, download redhat-virtualization-host-image-update-4.0-20161116.1.el7_3.noarch.rpm and update:
# yum install redhat-virtualization-host-image-update-4.0-20161116.1.el7_3.noarch.rpm
3. Reboot and login build2 rhvh-4.0-0.20161116.0, setup local repos, update to build3 redhat-virtualization-host-4.1-20170314.0:
# yum update
4. Reboot and focus on boot entry
5. Login redhat-virtualization-host-4.1-20170314.0, check files' group of /etc/ssh, ssh rhvh from other host.

Test results:
1. In step3, there is no error info during update "#yum update" and upgrade successful. After update,there are 2 bases/layers.
[root@dell-per730-35 ~]# imgbase layout
rhvh-4.0-0.20161116.0
 +- rhvh-4.0-0.20161116.0+1
rhvh-4.1-0.20170315.0
 +- rhvh-4.1-0.20170315.0+1
2. In step4, there are two latest build boot entries.
------------------------
rhvh-4.1-0.20170315.0
rhvh-4.0-0.20161116.0
------------------------
3. In step5, sshd service run normally, ssh rhvh from other machine successfully.
# ls -l /etc/ssh
total 276
-rw-r--r--. 1 root root     242153 Dec 20 16:27 moduli
-rw-r--r--. 1 root root       2208 Dec 20 16:27 ssh_config
-rw-------. 1 root root       4361 Dec 20 16:27 sshd_config
-rw-r-----. 1 root ssh_keys    227 Mar 16 03:24 ssh_host_ecdsa_key
-rw-r--r--. 1 root root        162 Mar 16 03:24 ssh_host_ecdsa_key.pub
-rw-r-----. 1 root ssh_keys    387 Mar 16 03:24 ssh_host_ed25519_key
-rw-r--r--. 1 root root         82 Mar 16 03:24 ssh_host_ed25519_key.pub
-rw-r-----. 1 root ssh_keys   1675 Mar 16 03:24 ssh_host_rsa_key
-rw-r--r--. 1 root root        382 Mar 16 03:24 ssh_host_rsa_key.pub

So this bug is fixed in imgbased-0.9.18-0.1.el7ev.noarch, change the status to VERIFIED.


Note You need to log in before you can comment on or make changes to this bug.