Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1594061

Summary: Yum install packages (e.g: wget/vim-enhanced) failed from rhel-7-server-rhvh-4* repos after upgraded RHVH.
Product: [oVirt] ovirt-node Reporter: Wei Wang <weiwang>
Component: UIAssignee: Ryan Barry <rbarry>
Status: CLOSED WORKSFORME QA Contact: Wei Wang <weiwang>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.2CC: bugs, cshao, huzhao, jiaczhan, qiyuan, rbarry, weiwang, yaniwang, ycui, yzhao
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-06-22 02:22:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Node RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
log files
none
repo config file none

Description Wei Wang 2018-06-22 01:12:51 UTC
Created attachment 1453628 [details]
log files

Description of problem:
Yum install some packages (e.g: wget/vim-enhanced) from rhel-7-server-rhvh-4* repos failed after upgraded RHVH. It is not reproduced without upgrade.
[root@dhcp-9-130 ~]# imgbase layout
rhvh-4.2.4.2-0.20180608.0
 +- rhvh-4.2.4.2-0.20180608.0+1
rhvh-4.2.4.3-0.20180615.0
 +- rhvh-4.2.4.3-0.20180615.0+1

[root@dhcp-9-130 ~]# yum install -y vim-enhanced
Loaded plugins: enabled_repos_upload, imgbased-persist, package_upload, product-id, search-disabled-repos, subscription-manager, vdsmupgrade
rhel-7-server-rhvh-4-beta-debug-rpms                                                                                                                                                    | 3.8 kB  00:00:00    
rhel-7-server-rhvh-4-beta-rpms                                                                                                                                                          | 3.8 kB  00:00:00    
Not using downloaded rhel-7-server-rhvh-4-beta-rpms/repomd.xml because it is older than what we have:
  Current   : Sun Jun 17 10:29:50 2018
  Downloaded: Mon Apr 16 23:27:13 2018
rhel-7-server-rhvh-4-beta-source-rpms                                                                                                                                                   | 3.8 kB  00:00:00    
rhel-7-server-rhvh-4-debug-rpms                                                                                                                                                         | 3.8 kB  00:00:00    
rhel-7-server-rhvh-4-rpms                                                                                                                                                               | 4.0 kB  00:00:00    
Not using downloaded rhel-7-server-rhvh-4-rpms/repomd.xml because it is older than what we have:
  Current   : Sun Jun 17 10:29:50 2018
  Downloaded: Fri Jun 15 03:24:13 2018
rhel-7-server-rhvh-4-source-rpms                                                                                                                                                        | 3.8 kB  00:00:00    
(1/12): rhel-7-server-rhvh-4-beta-debug-rpms/x86_64/group                                                                                                                               |  104 B  00:00:01    
(2/12): rhel-7-server-rhvh-4-beta-debug-rpms/x86_64/updateinfo                                                                                                                          |   76 B  00:00:01    
(3/12): rhel-7-server-rhvh-4-beta-debug-rpms/x86_64/primary_db                                                                                                                          |  10 kB  00:00:01    
(4/12): rhel-7-server-rhvh-4-beta-source-rpms/x86_64/group                                                                                                                              |  104 B  00:00:01    
(5/12): rhel-7-server-rhvh-4-beta-source-rpms/x86_64/updateinfo                                                                                                                         |   76 B  00:00:01    
(6/12): rhel-7-server-rhvh-4-beta-source-rpms/x86_64/primary_db                                                                                                                         |  14 kB  00:00:00    
(7/12): rhel-7-server-rhvh-4-debug-rpms/x86_64/group                                                                                                                                    |  104 B  00:00:01    
(8/12): rhel-7-server-rhvh-4-debug-rpms/x86_64/updateinfo                                                                                                                               |  23 kB  00:00:01    
(9/12): rhel-7-server-rhvh-4-debug-rpms/x86_64/primary_db                                                                                                                               |  13 kB  00:00:00    
(10/12): rhel-7-server-rhvh-4-source-rpms/x86_64/updateinfo                                                                                                                             |  26 kB  00:00:01    
(11/12): rhel-7-server-rhvh-4-source-rpms/x86_64/group                                                                                                                                  |  104 B  00:00:01    
(12/12): rhel-7-server-rhvh-4-source-rpms/x86_64/primary_db                                                                                                                             |  22 kB  00:00:01    
No package vim-enhanced available.
Error: Nothing to do
Uploading Enabled Repositories Report
Loaded plugins: product-id, subscription-manager
Unable to upload Enabled Repositories Report
[root@dhcp-9-130 ~]# yum install -y wget
Loaded plugins: enabled_repos_upload, imgbased-persist, package_upload, product-id, search-disabled-repos, subscription-manager, vdsmupgrade
No package wget available.
Error: Nothing to do
Uploading Enabled Repositories Report
Loaded plugins: product-id, subscription-manager
Unable to upload Enabled Repositories Report


Version-Release number of selected component (if applicable):
From
rhvh-4.2.4.2-0.20180608.0
to
rhvh-4.2.4.3-0.20180615.0

cockpit-system-169-1.el7.noarch
cockpit-ovirt-dashboard-0.11.28-1.el7ev.noarch
cockpit-169-1.el7.x86_64
cockpit-storaged-169-1.el7.noarch
cockpit-ws-169-1.el7.x86_64
cockpit-machines-ovirt-169-1.el7.noarch
cockpit-dashboard-169-1.el7.x86_64
cockpit-bridge-169-1.el7.x86_64
imgbased-1.0.19-0.1.el7ev.noarch
subscription-manager-1.20.11-1.el7_5.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Clean install rhvh-4.2.4.2-0.20180608.0
2. Upgrade to rhvh-4.2.4.3-0.20180615.0
3. Login to cockpit WebUI with root account on layer rhvh-4.2.4.3-0.20180615.0+1
4. Subscribe to RHSM in cockpit
5. Check the repos in RHVH
   #subscription-manager repos --list
6. Enable all the "rhel-7-server-rhvh-4*"
  #subscription-manager repos --enable=rhel-7-server-rhvh-4*
7. Yum install wget/vim-enhanced packages.

 
Actual results:
Install packages failed.


Expected results:
The packages should be installed successfully.


Additional info:
If the host has only one layer rhvh-4.2.4.3-0.20180615.0, this bug cannot be reproduced, the packages can be installed normally.

Comment 1 Ryan Barry 2018-06-22 01:29:51 UTC
Can you please try 'yum clean all'?

Comment 2 Wei Wang 2018-06-22 02:21:30 UTC
(In reply to Ryan Barry from comment #1)
> Can you please try 'yum clean all'?

Yes, after yum clean all, it is normal. Does it mean after upgrading via local repo file, the cache has something conflict with the subscribed repos? So we need to clean the cache?

Comment 3 Ryan Barry 2018-06-22 03:35:28 UTC
I'll check on the root cause.

We now also migrate individual files in /var, but this patch was not in 0615

From the output, it appears that there was time drift which invalidated the repo. Did some change in ntp or chrony happen?

>Not using downloaded rhel-7-server-rhvh-4-rpms/repomd.xml because it is older than what we have:

Comment 4 Wei Wang 2018-06-22 04:38:52 UTC
(In reply to Ryan Barry from comment #3)
> I'll check on the root cause.
> 
I will do the further test on it. For example only updating from rpm package, not the local repo config file.

> We now also migrate individual files in /var, but this patch was not in 0615

is it in 0608? 
> 
> From the output, it appears that there was time drift which invalidated the
> repo. Did some change in ntp or chrony happen?

No, I have not other operation within NTP and chrony.
> 
> >Not using downloaded rhel-7-server-rhvh-4-rpms/repomd.xml because it is older than what we have:

Comment 5 Ryan Barry 2018-06-22 04:43:51 UTC
No, the patch was earlier this week to fix vdsm configuration after upgrades:
https://gerrit.ovirt.org/92376

No changes prior to this have affected /var since early in 4.1

The only possible change here is in the RHEL batch set. Is it reproducible upgrading to today's build?

Comment 6 Wei Wang 2018-06-25 09:11:04 UTC
(In reply to Ryan Barry from comment #5)
> No, the patch was earlier this week to fix vdsm configuration after upgrades:
> https://gerrit.ovirt.org/92376
> 
> No changes prior to this have affected /var since early in 4.1
> 
> The only possible change here is in the RHEL batch set. Is it reproducible
> upgrading to today's build?

Retest this issue:
From
rhvh-4.2.4.2-0.20180608.0
To
rhvh-4.2.4.3-0.20180622.0

Scenario 1:
1. Clean install rhvh-4.2.4.2-0.20180608.0
2. Upgrade to rhvh-4.2.4.3-0.20180622.0 via yum install rpm package.
3. Login to cockpit WebUI with root account on layer rhvh-4.2.4.3-0.20180622.0+1
4. Subscribe to RHSM in the cockpit
5. Check the repos in RHVH
   #subscription-manager repos --list
6. Enable all the "rhel-7-server-rhvh-4*"
  #subscription-manager repos --enable=rhel-7-server-rhvh-4*
7. Yum install wget/vim-enhanced packages successfully

Scenario 2:
1. Clean install rhvh-4.2.4.2-0.20180608.0
2. Upgrade to rhvh-4.2.4.3-0.20180622.0 via local repo config file.
3. Login to cockpit WebUI with root account on layer rhvh-4.2.4.3-0.20180622.0+1
4. Subscribe to RHSM in the cockpit
5. Check the repos in RHVH
   #subscription-manager repos --list
6. Enable all the "rhel-7-server-rhvh-4*"
  #subscription-manager repos --enable=rhel-7-server-rhvh-4*
7. Yum install wget/vim-enhanced packages failed.
8. Yum clean all
9. Yum install wget/vim-enhanced packages successfully.

So I think it maybe related to my local repo config file, attach it into attachment.

Comment 7 Wei Wang 2018-06-25 09:11:47 UTC
Created attachment 1454296 [details]
repo config file