Created attachment 1258310 [details] /tmp/imgbased.log Description of problem: Upgrade from wrapper to wrapper failed: 1. There is error info during "yum update" 2. There is 3 bases/layers, which should be 2 bases/layers 3. Miss boot entry of latest build Please see /tmp/imgbased.log, you'd see that rsync is running out of space which raises a ConfigMigrationError. Test version: Build1: redhat-virtualization-host-4.0-20160919.0 Build2: redhat-virtualization-host-4.0-20161116.0 Build3: redhat-virtualization-host-4.1-20170222.0 # imgbase layout rhvh-4.0-0.20160919.0 +- rhvh-4.0-0.20160919.0+1 rhvh-4.0-0.20161116.0 +- rhvh-4.0-0.20161116.0+1 rhvh-4.1-0.20170223.0 +- rhvh-4.1-0.20170223.0+1 Test steps: 1. Install build1 redhat-virtualization-host-4.0-20160919.0 2. Reboot and login build1, download redhat-virtualization-host-image-update-4.0-20161116.1.el7_3.noarch.rpm and update: # yum install redhat-virtualization-host-image-update-4.0-20161116.1.el7_3.noarch.rpm 3. Reboot and login build2 rhvh-4.0-0.20161116.0, setup local repos, update to build3 redhat-virtualization-host-4.1-20170222.0: # yum update 4. Reboot and focus on boot entry Actual results: 1. In step3, there is error info during update "#yum update". After update,there are 3 bases. # yum update Loaded plugins: imgbased-warning, product-id, search-disabled-repos, subscription-manager This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. Warning: yum operations are not persisted across upgrades! Repo rhel-7-server-rhvh-4-source-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877.pem Repo rhel-7-server-rhvh-4-source-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877-key.pem Repo rhel-7-server-rhvh-4-beta-debug-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877.pem Repo rhel-7-server-rhvh-4-beta-debug-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877-key.pem Repo rhel-7-server-rhvh-4-debug-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877.pem Repo rhel-7-server-rhvh-4-debug-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877-key.pem Repo rhel-7-server-rhvh-4-beta-source-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877.pem Repo rhel-7-server-rhvh-4-beta-source-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877-key.pem Repo rhel-7-server-rhvh-4-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877.pem Repo rhel-7-server-rhvh-4-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877-key.pem Repo rhel-7-server-rhvh-4-beta-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877.pem Repo rhel-7-server-rhvh-4-beta-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877-key.pem rhel-7-server-rhvh-4-beta-debug-rpms | 2.9 kB 00:00:00 rhel-7-server-rhvh-4-beta-rpms | 2.9 kB 00:00:00 rhel-7-server-rhvh-4-beta-source-rpms | 2.9 kB 00:00:00 rhel-7-server-rhvh-4-debug-rpms | 2.9 kB 00:00:00 rhel-7-server-rhvh-4-rpms | 2.9 kB 00:00:00 rhel-7-server-rhvh-4-source-rpms | 2.9 kB 00:00:00 (1/6): rhel-7-server-rhvh-4-beta-debug-rpms/primary_db | 1.7 kB 00:00:00 (2/6): rhel-7-server-rhvh-4-source-rpms/primary_db | 4.1 kB 00:00:00 (3/6): rhel-7-server-rhvh-4-beta-rpms/primary_db | 9.3 kB 00:00:00 (4/6): rhel-7-server-rhvh-4-debug-rpms/primary_db | 1.7 kB 00:00:00 (5/6): rhel-7-server-rhvh-4-rpms/primary_db | 9.3 kB 00:00:00 (6/6): rhel-7-server-rhvh-4-beta-source-rpms/primary_db | 4.1 kB 00:00:00 Resolving Dependencies --> Running transaction check ---> Package redhat-virtualization-host-image-update.noarch 0:4.1-20170222.0.el7_3 will be obsoleting ---> Package redhat-virtualization-host-image-update-placeholder.noarch 0:4.0-5.2.el7 will be obsoleted --> Finished Dependency Resolution Dependencies Resolved ====================================================================================================================================================================== Package Arch Version Repository Size ====================================================================================================================================================================== Installing: redhat-virtualization-host-image-update noarch 4.1-20170222.0.el7_3 rhel-7-server-rhvh-4-beta-rpms 548 M replacing redhat-virtualization-host-image-update-placeholder.noarch 4.0-5.2.el7 Transaction Summary ====================================================================================================================================================================== Install 1 Package Total download size: 548 M Is this ok [y/d/N]: y Downloading packages: redhat-virtualization-host-image-update-latest.rpm | 548 MB 00:00:07 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : redhat-virtualization-host-image-update-4.1-20170222.0.el7_3.noarch 1/2 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 cp: warning: source file ‘/var/cache/yum/x86_64/7RedHatVirtualizationHost/rhel-7-server-rhvh-4-beta-rpms/packages/redhat-virtualization-host-image-update-latest.rpm’ specified more than once Erasing : redhat-virtualization-host-image-update-placeholder-4.0-5.2.el7.noarch 2/2 Verifying : redhat-virtualization-host-image-update-4.1-20170222.0.el7_3.noarch 1/2 Verifying : redhat-virtualization-host-image-update-placeholder-4.0-5.2.el7.noarch 2/2 Installed: redhat-virtualization-host-image-update.noarch 0:4.1-20170222.0.el7_3 Replaced: redhat-virtualization-host-image-update-placeholder.noarch 0:4.0-5.2.el7 Complete! [root@dhcp-10-16 ~]# imgbase w [INFO] You are on rhvh-4.0-0.20161116.0+1 [root@dhcp-10-16 ~]# imgbase layout rhvh-4.0-0.20160919.0 +- rhvh-4.0-0.20160919.0+1 rhvh-4.0-0.20161116.0 +- rhvh-4.0-0.20161116.0+1 rhvh-4.1-0.20170223.0 +- rhvh-4.1-0.20170223.0+1 2. In step4, there is not boot entry of build3 rhvh-4.1-0.20170223.0, only 2 boot entry of rhvh-4.0-0.20160919.0 and rhvh-4.0-0.20161116.0. And login rhvh-4.0-0.20161116.0, encounter Bug 1426151. Expected results: 1. In step3, there should be no error info during update. After upgrade, there should be only 2 latest bases. 2. In step4, there should be boot entry of rhvh-4.1-0.20170223.0 and rhvh-4.0-0.20161116.0 Additional info:
*** Bug 1427478 has been marked as a duplicate of this bug. ***
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.
I just tried updating: 4.0-20160916.0 4.0-20161116.1 4.1-20170308.1 Still can't reproduce... Another reproducer would be great, if possible.
huzhao, Could you help to reply #c5? Thanks.
(In reply to Ryan Barry from comment #5) > I just tried updating: > > 4.0-20160916.0 > 4.0-20161116.1 > 4.1-20170308.1 > > Still can't reproduce... > > Another reproducer would be great, if possible. Ryan, I reproduced again with another machine. 4.0-20160919.0 4.0-20161116.1 4.1-20170308.1 I modified /etc/ssh/ssh_host_ed25519_key to /etc/ssh/ssh_host_ed25519_key_backup, then can ssh successful. Please refer to attachment for log. And I will send ENV info via email.
Created attachment 1262772 [details] comment 7 : sosreport and log
I tried to reproduce this issue on my local testing machine, and it can be reproduced. The three builds are the same as comment #7. After upgrade to 4.1-20170308.1, I checked sshd.service and /etc/ssh: # systemctl status sshd ● sshd.service - OpenSSH server daemon Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2017-03-14 14:18:16 CST; 1min 1s ago Docs: man:sshd(8) man:sshd_config(5) Process: 21658 ExecStart=/usr/sbin/sshd $OPTIONS (code=exited, status=0/SUCCESS) Main PID: 21660 (sshd) CGroup: /system.slice/sshd.service └─21660 /usr/sbin/sshd Mar 14 14:18:16 dhcp-9-139.nay.redhat.com sshd[21658]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ Mar 14 14:18:16 dhcp-9-139.nay.redhat.com sshd[21658]: Permissions 0640 for '/etc/ssh/ssh_host_ed25519_key' are too open. Mar 14 14:18:16 dhcp-9-139.nay.redhat.com sshd[21658]: It is required that your private key files are NOT accessible by others. Mar 14 14:18:16 dhcp-9-139.nay.redhat.com sshd[21658]: This private key will be ignored. Mar 14 14:18:16 dhcp-9-139.nay.redhat.com sshd[21658]: bad permissions: ignore key: /etc/ssh/ssh_host_ed25519_key Mar 14 14:18:16 dhcp-9-139.nay.redhat.com sshd[21658]: Could not load host key: /etc/ssh/ssh_host_ed25519_key Mar 14 14:18:16 dhcp-9-139.nay.redhat.com systemd[1]: PID file /var/run/sshd.pid not readable (yet?) after start. Mar 14 14:18:16 dhcp-9-139.nay.redhat.com sshd[21660]: Server listening on 0.0.0.0 port 22. Mar 14 14:18:16 dhcp-9-139.nay.redhat.com systemd[1]: Started OpenSSH server daemon. Mar 14 14:18:16 dhcp-9-139.nay.redhat.com sshd[21660]: Server listening on :: port 22. # ls -al /etc/ssh drwxr-xr-x. 2 root root 225 Mar 14 14:06 . drwxr-xr-x. 118 root root 8192 Mar 14 14:16 .. -rw-r--r--. 1 root root 242153 Dec 21 00:27 moduli -rw-r--r--. 1 root root 2208 Dec 21 00:27 ssh_config -rw-------. 1 root root 4361 Dec 21 00:27 sshd_config -rw-r-----. 1 root input 227 Mar 14 05:17 ssh_host_ecdsa_key -rw-r--r--. 1 root root 162 Mar 14 05:17 ssh_host_ecdsa_key.pub -rw-r-----. 1 root input 387 Mar 14 05:17 ssh_host_ed25519_key -rw-r--r--. 1 root root 82 Mar 14 05:17 ssh_host_ed25519_key.pub -rw-r-----. 1 root input 1675 Mar 14 05:17 ssh_host_rsa_key -rw-r--r--. 1 root root 382 Mar 14 05:17 ssh_host_rsa_key.pub The group of ssh_host_ecdsa_key, ssh_host_ed25519_key, and ssh_host_rsa_key is input. This is incorrect, it should be ssh_keys. I changed the group to ssh_keys, using cmd: # chown root:ssh_keys ssh_host_ecdsa_key # chown root:ssh_keys ssh_host_ed25519_key # chown root:ssh_keys ssh_host_rsa_key and restart sshd using cmd: # systemctl restart sshd recheck sshd.service: # systemctl status sshd ● sshd.service - OpenSSH server daemon Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2017-03-14 14:22:09 CST; 17s ago Docs: man:sshd(8) man:sshd_config(5) Process: 21698 ExecStart=/usr/sbin/sshd $OPTIONS (code=exited, status=0/SUCCESS) Main PID: 21700 (sshd) CGroup: /system.slice/sshd.service └─21700 /usr/sbin/sshd Mar 14 14:22:09 dhcp-9-139.nay.redhat.com systemd[1]: Starting OpenSSH server daemon... Mar 14 14:22:09 dhcp-9-139.nay.redhat.com systemd[1]: PID file /var/run/sshd.pid not readable (yet?) after start. Mar 14 14:22:09 dhcp-9-139.nay.redhat.com sshd[21700]: Server listening on 0.0.0.0 port 22. Mar 14 14:22:09 dhcp-9-139.nay.redhat.com sshd[21700]: Server listening on :: port 22. Mar 14 14:22:09 dhcp-9-139.nay.redhat.com systemd[1]: Started OpenSSH server daemon. Finally, this issue has been resloved by this workaroud, sshd returns to normal status.
Test version: Build1: redhat-virtualization-host-4.0-20160919.0 Build2: redhat-virtualization-host-4.0-20161116.0 Build3: redhat-virtualization-host-4.1-20170314.0 imgbased-0.9.18-0.1.el7ev.noarch Test steps: 1. Install build1 redhat-virtualization-host-4.0-20160919.0 2. Reboot and login build1, download redhat-virtualization-host-image-update-4.0-20161116.1.el7_3.noarch.rpm and update: # yum install redhat-virtualization-host-image-update-4.0-20161116.1.el7_3.noarch.rpm 3. Reboot and login build2 rhvh-4.0-0.20161116.0, setup local repos, update to build3 redhat-virtualization-host-4.1-20170314.0: # yum update 4. Reboot and focus on boot entry 5. Login redhat-virtualization-host-4.1-20170314.0, check files' group of /etc/ssh, ssh rhvh from other host. Test results: 1. In step3, there is no error info during update "#yum update" and upgrade successful. After update,there are 2 bases/layers. [root@dell-per730-35 ~]# imgbase layout rhvh-4.0-0.20161116.0 +- rhvh-4.0-0.20161116.0+1 rhvh-4.1-0.20170315.0 +- rhvh-4.1-0.20170315.0+1 2. In step4, there are two latest build boot entries. ------------------------ rhvh-4.1-0.20170315.0 rhvh-4.0-0.20161116.0 ------------------------ 3. In step5, sshd service run normally, ssh rhvh from other machine successfully. # ls -l /etc/ssh total 276 -rw-r--r--. 1 root root 242153 Dec 20 16:27 moduli -rw-r--r--. 1 root root 2208 Dec 20 16:27 ssh_config -rw-------. 1 root root 4361 Dec 20 16:27 sshd_config -rw-r-----. 1 root ssh_keys 227 Mar 16 03:24 ssh_host_ecdsa_key -rw-r--r--. 1 root root 162 Mar 16 03:24 ssh_host_ecdsa_key.pub -rw-r-----. 1 root ssh_keys 387 Mar 16 03:24 ssh_host_ed25519_key -rw-r--r--. 1 root root 82 Mar 16 03:24 ssh_host_ed25519_key.pub -rw-r-----. 1 root ssh_keys 1675 Mar 16 03:24 ssh_host_rsa_key -rw-r--r--. 1 root root 382 Mar 16 03:24 ssh_host_rsa_key.pub So this bug is fixed in imgbased-0.9.18-0.1.el7ev.noarch, change the status to VERIFIED.