Created attachment 1453628 [details] log files Description of problem: Yum install some packages (e.g: wget/vim-enhanced) from rhel-7-server-rhvh-4* repos failed after upgraded RHVH. It is not reproduced without upgrade. [root@dhcp-9-130 ~]# imgbase layout rhvh-4.2.4.2-0.20180608.0 +- rhvh-4.2.4.2-0.20180608.0+1 rhvh-4.2.4.3-0.20180615.0 +- rhvh-4.2.4.3-0.20180615.0+1 [root@dhcp-9-130 ~]# yum install -y vim-enhanced Loaded plugins: enabled_repos_upload, imgbased-persist, package_upload, product-id, search-disabled-repos, subscription-manager, vdsmupgrade rhel-7-server-rhvh-4-beta-debug-rpms | 3.8 kB 00:00:00 rhel-7-server-rhvh-4-beta-rpms | 3.8 kB 00:00:00 Not using downloaded rhel-7-server-rhvh-4-beta-rpms/repomd.xml because it is older than what we have: Current : Sun Jun 17 10:29:50 2018 Downloaded: Mon Apr 16 23:27:13 2018 rhel-7-server-rhvh-4-beta-source-rpms | 3.8 kB 00:00:00 rhel-7-server-rhvh-4-debug-rpms | 3.8 kB 00:00:00 rhel-7-server-rhvh-4-rpms | 4.0 kB 00:00:00 Not using downloaded rhel-7-server-rhvh-4-rpms/repomd.xml because it is older than what we have: Current : Sun Jun 17 10:29:50 2018 Downloaded: Fri Jun 15 03:24:13 2018 rhel-7-server-rhvh-4-source-rpms | 3.8 kB 00:00:00 (1/12): rhel-7-server-rhvh-4-beta-debug-rpms/x86_64/group | 104 B 00:00:01 (2/12): rhel-7-server-rhvh-4-beta-debug-rpms/x86_64/updateinfo | 76 B 00:00:01 (3/12): rhel-7-server-rhvh-4-beta-debug-rpms/x86_64/primary_db | 10 kB 00:00:01 (4/12): rhel-7-server-rhvh-4-beta-source-rpms/x86_64/group | 104 B 00:00:01 (5/12): rhel-7-server-rhvh-4-beta-source-rpms/x86_64/updateinfo | 76 B 00:00:01 (6/12): rhel-7-server-rhvh-4-beta-source-rpms/x86_64/primary_db | 14 kB 00:00:00 (7/12): rhel-7-server-rhvh-4-debug-rpms/x86_64/group | 104 B 00:00:01 (8/12): rhel-7-server-rhvh-4-debug-rpms/x86_64/updateinfo | 23 kB 00:00:01 (9/12): rhel-7-server-rhvh-4-debug-rpms/x86_64/primary_db | 13 kB 00:00:00 (10/12): rhel-7-server-rhvh-4-source-rpms/x86_64/updateinfo | 26 kB 00:00:01 (11/12): rhel-7-server-rhvh-4-source-rpms/x86_64/group | 104 B 00:00:01 (12/12): rhel-7-server-rhvh-4-source-rpms/x86_64/primary_db | 22 kB 00:00:01 No package vim-enhanced available. Error: Nothing to do Uploading Enabled Repositories Report Loaded plugins: product-id, subscription-manager Unable to upload Enabled Repositories Report [root@dhcp-9-130 ~]# yum install -y wget Loaded plugins: enabled_repos_upload, imgbased-persist, package_upload, product-id, search-disabled-repos, subscription-manager, vdsmupgrade No package wget available. Error: Nothing to do Uploading Enabled Repositories Report Loaded plugins: product-id, subscription-manager Unable to upload Enabled Repositories Report Version-Release number of selected component (if applicable): From rhvh-4.2.4.2-0.20180608.0 to rhvh-4.2.4.3-0.20180615.0 cockpit-system-169-1.el7.noarch cockpit-ovirt-dashboard-0.11.28-1.el7ev.noarch cockpit-169-1.el7.x86_64 cockpit-storaged-169-1.el7.noarch cockpit-ws-169-1.el7.x86_64 cockpit-machines-ovirt-169-1.el7.noarch cockpit-dashboard-169-1.el7.x86_64 cockpit-bridge-169-1.el7.x86_64 imgbased-1.0.19-0.1.el7ev.noarch subscription-manager-1.20.11-1.el7_5.x86_64 How reproducible: 100% Steps to Reproduce: 1. Clean install rhvh-4.2.4.2-0.20180608.0 2. Upgrade to rhvh-4.2.4.3-0.20180615.0 3. Login to cockpit WebUI with root account on layer rhvh-4.2.4.3-0.20180615.0+1 4. Subscribe to RHSM in cockpit 5. Check the repos in RHVH #subscription-manager repos --list 6. Enable all the "rhel-7-server-rhvh-4*" #subscription-manager repos --enable=rhel-7-server-rhvh-4* 7. Yum install wget/vim-enhanced packages. Actual results: Install packages failed. Expected results: The packages should be installed successfully. Additional info: If the host has only one layer rhvh-4.2.4.3-0.20180615.0, this bug cannot be reproduced, the packages can be installed normally.
Can you please try 'yum clean all'?
(In reply to Ryan Barry from comment #1) > Can you please try 'yum clean all'? Yes, after yum clean all, it is normal. Does it mean after upgrading via local repo file, the cache has something conflict with the subscribed repos? So we need to clean the cache?
I'll check on the root cause. We now also migrate individual files in /var, but this patch was not in 0615 From the output, it appears that there was time drift which invalidated the repo. Did some change in ntp or chrony happen? >Not using downloaded rhel-7-server-rhvh-4-rpms/repomd.xml because it is older than what we have:
(In reply to Ryan Barry from comment #3) > I'll check on the root cause. > I will do the further test on it. For example only updating from rpm package, not the local repo config file. > We now also migrate individual files in /var, but this patch was not in 0615 is it in 0608? > > From the output, it appears that there was time drift which invalidated the > repo. Did some change in ntp or chrony happen? No, I have not other operation within NTP and chrony. > > >Not using downloaded rhel-7-server-rhvh-4-rpms/repomd.xml because it is older than what we have:
No, the patch was earlier this week to fix vdsm configuration after upgrades: https://gerrit.ovirt.org/92376 No changes prior to this have affected /var since early in 4.1 The only possible change here is in the RHEL batch set. Is it reproducible upgrading to today's build?
(In reply to Ryan Barry from comment #5) > No, the patch was earlier this week to fix vdsm configuration after upgrades: > https://gerrit.ovirt.org/92376 > > No changes prior to this have affected /var since early in 4.1 > > The only possible change here is in the RHEL batch set. Is it reproducible > upgrading to today's build? Retest this issue: From rhvh-4.2.4.2-0.20180608.0 To rhvh-4.2.4.3-0.20180622.0 Scenario 1: 1. Clean install rhvh-4.2.4.2-0.20180608.0 2. Upgrade to rhvh-4.2.4.3-0.20180622.0 via yum install rpm package. 3. Login to cockpit WebUI with root account on layer rhvh-4.2.4.3-0.20180622.0+1 4. Subscribe to RHSM in the cockpit 5. Check the repos in RHVH #subscription-manager repos --list 6. Enable all the "rhel-7-server-rhvh-4*" #subscription-manager repos --enable=rhel-7-server-rhvh-4* 7. Yum install wget/vim-enhanced packages successfully Scenario 2: 1. Clean install rhvh-4.2.4.2-0.20180608.0 2. Upgrade to rhvh-4.2.4.3-0.20180622.0 via local repo config file. 3. Login to cockpit WebUI with root account on layer rhvh-4.2.4.3-0.20180622.0+1 4. Subscribe to RHSM in the cockpit 5. Check the repos in RHVH #subscription-manager repos --list 6. Enable all the "rhel-7-server-rhvh-4*" #subscription-manager repos --enable=rhel-7-server-rhvh-4* 7. Yum install wget/vim-enhanced packages failed. 8. Yum clean all 9. Yum install wget/vim-enhanced packages successfully. So I think it maybe related to my local repo config file, attach it into attachment.
Created attachment 1454296 [details] repo config file