Description of problem: Example from https://build.gluster.org/job/devrpm-el6/18900/console : INFO: Start(glusterfs-8dev-0.355.gitcd6fc44.el7.src.rpm) Config(epel-6-x86_64) Start: clean chroot Finish: clean chroot Start: chroot init INFO: calling preinit hooks INFO: enabled root cache Start: unpacking root cache Finish: unpacking root cache INFO: enabled yum cache Start: cleaning yum metadata Finish: cleaning yum metadata INFO: enabled HW Info plugin Mock Version: 1.4.21 INFO: Mock Version: 1.4.21 Start: yum update ERROR: Exception(glusterfs-8dev-0.355.gitcd6fc44.el7.src.rpm) Config(epel-6-x86_64) 2 minutes 47 seconds INFO: Results and/or logs in: /home/jenkins/root/workspace/devrpm-el6/RPMS/el6/x86_64/ INFO: Cleaning up build root ('cleanup_on_failure=True') Seems to me like a confusion between EL7 RPMs and EL6 environment. Looking at the debug log (https://build.gluster.org/job/devrpm-el6/18900/artifact/RPMS/el6/x86_64/root.log), I see this: DEBUG util.py:218: creating dir: /var/lib/mock/epel-6-x86_64/rpmconfig DEBUG package_manager.py:128: ['/usr/bin/yum', '--installroot', '/var/lib/mock/epel-6-x86_64/root/', '-y', '--releasever', '6', 'update'] DEBUG util.py:763: child environment: None DEBUG util.py:689: Executing command: ['/usr/bin/yum', '--installroot', '/var/lib/mock/epel-6-x86_64/root/', '-y', '--releasever', '6', 'update', '--setopt=tsflags=nocontexts'] with env {'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOME': '/var/lib/mock/epel-6-x86_64/rpmconfig', 'HOSTNAME': 'mock', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'PROMPT_COMMAND': 'printf "\\033]0;<mock-chroot>\\007"', 'PS1': '<mock-chroot> \\s-\\v\\$ ', 'LANG': 'en_US.UTF-8', 'LC_MESSAGES': 'C.UTF-8'} and shell False DEBUG util.py:596: Failed to set locale, defaulting to C DEBUG util.py:596: Not using downloaded epel/repomd.xml because it is older than what we have: DEBUG util.py:596: Current : Sat Nov 30 01:47:11 2019 DEBUG util.py:596: Downloaded: Mon Nov 4 00:18:04 2019 DEBUG util.py:596: http://mirror.us-midwest-1.nexcess.net/epel/6/x86_64/repodata/4a71567d42bbff26febf25c97792d0ffd049438e2fada1286a56f05ebc32b75b-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found DEBUG util.py:596: Trying other mirror. DEBUG util.py:596: To address this issue please refer to the below wiki article DEBUG util.py:596: https://wiki.centos.org/yum-errors DEBUG util.py:596: If above article doesn't help to resolve this issue please use https://bugs.centos.org/. DEBUG util.py:596: http://mirror.umd.edu/fedora/epel/6/x86_64/repodata/4a71567d42bbff26febf25c97792d0ffd049438e2fada1286a56f05ebc32b75b-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found DEBUG util.py:596: Trying other mirror.
Should we consider to drop this job as a part of smoke testing? I don't think we need it any longer.
I'm fine with it - but I'm not sure if we've decided to drop EL6 support upstream? (I know we did downstream). Niels - are you still building for CentOS 6?
(In reply to Yaniv Kaul from comment #2) > I'm fine with it - but I'm not sure if we've decided to drop EL6 support > upstream? (I know we did downstream). > Niels - are you still building for CentOS 6? We are still building the client-side parts for CentOS 6, server parts are for CentOS 7 (and soon CentOS 8). The error that is hit looks like there is/was some mirror providing incorrect contents. The cached contents might be broken, and because it is considered newer than what is available on other mirrors, it does not get refreshed. Deleting the caches for mock might solve the issues (or maybe there is some timeout after which caches always get refreshed?).
I did tried to debug yesterday afternoon. Rebooting didn't change much (in case of some strange in memory cache), removing the chroot several time didn't work either. And running the command by hand did work, so I suspect there is a cache somewhere that I did miss on the mock side. That builder is not using the internal proxy, so that can be ruled out afaik. In the mean time, deepshika did remove the builderfrom rotation, I should have done that but I needed it to test the job each time, and forgot when i left out for the night.
ok so on closer mock --help reading, I did notice that --clean and --scrub are different commands, so I ran : "mock --scrub all -r epel-6-x86_64" and I am testing now on https://build.gluster.org/job/devrpm-el6/18944/console
And that's look like a success. So TIL, --clean is different from --scrub. Guess that explain the state of my kitchen. I did start to look at the script we run, but I am not sure if we should scrub each time, since this consume more ressources on build. If the error come again too often, we might need to fix that, but in the mean time, let's keep it this way.