Bug 1787196

Summary:	EL6 smoke tests are failing
Product:	[Community] GlusterFS	Reporter:	Yaniv Kaul <ykaul>
Component:	project-infrastructure	Assignee:	bugs <bugs>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	mainline	CC:	bugs, dkhandel, gluster-infra, mscherer, ndevos, pasik
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-02-19 06:57:14 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Yaniv Kaul 2020-01-01 10:58:17 UTC

Description of problem:
Example from https://build.gluster.org/job/devrpm-el6/18900/console :
INFO: Start(glusterfs-8dev-0.355.gitcd6fc44.el7.src.rpm)  Config(epel-6-x86_64)
Start: clean chroot
Finish: clean chroot
Start: chroot init
INFO: calling preinit hooks
INFO: enabled root cache
Start: unpacking root cache
Finish: unpacking root cache
INFO: enabled yum cache
Start: cleaning yum metadata
Finish: cleaning yum metadata
INFO: enabled HW Info plugin
Mock Version: 1.4.21
INFO: Mock Version: 1.4.21
Start: yum update
ERROR: Exception(glusterfs-8dev-0.355.gitcd6fc44.el7.src.rpm) Config(epel-6-x86_64) 2 minutes 47 seconds
INFO: Results and/or logs in: /home/jenkins/root/workspace/devrpm-el6/RPMS/el6/x86_64/
INFO: Cleaning up build root ('cleanup_on_failure=True')

Seems to me like a confusion between EL7 RPMs and EL6 environment.

Looking at the debug log (https://build.gluster.org/job/devrpm-el6/18900/artifact/RPMS/el6/x86_64/root.log), I see this:

DEBUG util.py:218:  creating dir: /var/lib/mock/epel-6-x86_64/rpmconfig
DEBUG package_manager.py:128:  ['/usr/bin/yum', '--installroot', '/var/lib/mock/epel-6-x86_64/root/', '-y', '--releasever', '6', 'update']
DEBUG util.py:763:  child environment: None
DEBUG util.py:689:  Executing command: ['/usr/bin/yum', '--installroot', '/var/lib/mock/epel-6-x86_64/root/', '-y', '--releasever', '6', 'update', '--setopt=tsflags=nocontexts'] with env {'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOME': '/var/lib/mock/epel-6-x86_64/rpmconfig', 'HOSTNAME': 'mock', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'PROMPT_COMMAND': 'printf "\\033]0;<mock-chroot>\\007"', 'PS1': '<mock-chroot> \\s-\\v\\$ ', 'LANG': 'en_US.UTF-8', 'LC_MESSAGES': 'C.UTF-8'} and shell False
DEBUG util.py:596:  Failed to set locale, defaulting to C
DEBUG util.py:596:  Not using downloaded epel/repomd.xml because it is older than what we have:
DEBUG util.py:596:    Current   : Sat Nov 30 01:47:11 2019
DEBUG util.py:596:    Downloaded: Mon Nov  4 00:18:04 2019
DEBUG util.py:596:  http://mirror.us-midwest-1.nexcess.net/epel/6/x86_64/repodata/4a71567d42bbff26febf25c97792d0ffd049438e2fada1286a56f05ebc32b75b-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found
DEBUG util.py:596:  Trying other mirror.
DEBUG util.py:596:  To address this issue please refer to the below wiki article 
DEBUG util.py:596:  https://wiki.centos.org/yum-errors
DEBUG util.py:596:  If above article doesn't help to resolve this issue please use https://bugs.centos.org/.
DEBUG util.py:596:  http://mirror.umd.edu/fedora/epel/6/x86_64/repodata/4a71567d42bbff26febf25c97792d0ffd049438e2fada1286a56f05ebc32b75b-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found
DEBUG util.py:596:  Trying other mirror.

Comment 1 Deepshikha khandelwal 2020-01-03 06:44:37 UTC

Should we consider to drop this job as a part of smoke testing? I don't think we need it any longer.

Comment 2 Yaniv Kaul 2020-01-03 07:00:44 UTC

I'm fine with it - but I'm not sure if we've decided to drop EL6 support upstream? (I know we did downstream).
Niels - are you still building for CentOS 6?

Comment 3 Niels de Vos 2020-01-03 10:32:18 UTC

(In reply to Yaniv Kaul from comment #2)
> I'm fine with it - but I'm not sure if we've decided to drop EL6 support
> upstream? (I know we did downstream).
> Niels - are you still building for CentOS 6?

We are still building the client-side parts for CentOS 6, server parts are for CentOS 7 (and soon CentOS 8).


The error that is hit looks like there is/was some mirror providing incorrect contents. The cached contents might be broken, and because it is considered newer than what is available on other mirrors, it does not get refreshed. Deleting the caches for mock might solve the issues (or maybe there is some timeout after which caches always get refreshed?).

Comment 4 M. Scherer 2020-01-03 12:23:17 UTC

I did tried to debug yesterday afternoon. Rebooting didn't change much (in case of some strange in memory cache), removing the chroot several time didn't work either. And running the command by hand did work, so I suspect there is a cache somewhere that I did miss on the mock side. That builder is not using the internal proxy, so that can be ruled out afaik.

In the mean time, deepshika did remove the builderfrom rotation, I should have done that but I needed it to test the job each time, and forgot when i left out for the night.

Comment 5 M. Scherer 2020-01-03 12:29:00 UTC

ok so on closer mock --help reading, I did notice that --clean and --scrub are different commands, so I ran : "mock --scrub all -r epel-6-x86_64" and I am testing now on https://build.gluster.org/job/devrpm-el6/18944/console

Comment 6 M. Scherer 2020-01-03 15:15:37 UTC

And that's look like a success. So TIL, --clean is different from --scrub. Guess that explain the state of my kitchen.

I did start to look at the script we run, but I am not sure if we should scrub each time, since this consume more ressources on build. If the error come again too often, we might need to fix that, but in the mean time, let's keep it this way.