Description of problem: Our local mirror of the epel/8/Everything/{x86_64,aarch64}/ repos started pulling a repomd.xml containing nonexistent location hrefs earlier this week. The problem persists after multiple successful rsync cycles over multiple days. This results in dnf reporting failure to download the non-existent xml during metadata update. This is occurring for our current mirrors of EPEL 8 and 9 "Everything" repos but does not occur for the "Modular" repos. Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: 1. Rsync with: rsync -rvlptDSH --exclude="*.~tmp~" --delete-delay --delay-updates rsync://mirror.umd.edu/fedora/epel/ /var/repo/epel/ >> "$logfile" 2>&1 2. chown and chmod the tree as usual for nginx read access 3. On a downstream client: 'dnf clean all', 'dnf makecache' Actual results: 07:58:32 root:~ # dnf clean all 89 files removed 07:58:37 root:~ # dnf makecache AuriStorFS for EL8 7.7 MB/s | 59 kB 00:00 CMF Servers for EL8 2.6 MB/s | 11 kB 00:00 CMF Servers for EL8 - NoArch 351 kB/s | 1.4 kB 00:00 CMF Servers for EL8 - Source 323 kB/s | 1.2 kB 00:00 Extra Packages for Enterprise Linux 8 12 MB/s | 130 kB 00:00 Errors during downloading metadata for repository 'epel': - Status code: 404 for http://admin-eth0/epel/8/Everything/aarch64/repodata/a032b1c806230cb2459b560619c336701b1fdfff664ab368c56309c5bf9292fc-filelists.xml.gz (IP: 10.1.0.1) - Status code: 404 for http://admin-eth0/epel/8/Everything/aarch64/repodata/4c4f92104a1abdd848086877c187f83c0fae4f63a929e56fd98f10c6af2fe191-primary.xml.gz (IP: 10.1.0.1) Error: Failed to download metadata for repo 'epel': Yum repo downloading error: Downloading error(s): repodata/4c4f92104a1abdd848086877c187f83c0fae4f63a929e56fd98f10c6af2fe191-primary.xml.gz - Cannot download, all mirrors were already tried without success; repodata/a032b1c806230cb2459b560619c336701b1fdfff664ab368c56309c5bf9292fc-filelists.xml.gz - Cannot download, all mirrors were already tried without success; repodata/f25d45e0aaf63956851c221a41843beb36b17ee4d29bf600b30f3a03ab34bf2a-prestodelta.xml.xz - Cannot download, all mirrors were already tried without success; repodata/f6504da72d77acc6b31db0ec0077bc580b424d3e5599f4e94bd2eb59dfd9684a-updateinfo.xml.bz2 - Cannot download, all mirrors were already tried without success Expected results: Download of location hrefs per upstream EPEL mirror repodata/repomd.xml Additional info: # One-shot demonstrator: 07:56:27 root:~ # cat check_repomd_xml_locations #!/bin/bash if [ -r "$1/repodata/repomd.xml" ] ;then grep 'location href' "$1"/repodata/repomd.xml |while read line ;do location="$(echo "$line" |awk 'BEGIN{FS="\""};{print $2}')" [ -r "$1/$location" ] && echo "ok: $location" || echo "fail: $location" done else echo "unable to read $1/repodata/repomd.xml" exit 1 fi # Results: 08:00:54 root:~ # ./check_repomd_xml_locations /var/repo/epel/8/Everything/x86_64/ fail: repodata/8f4ab96dc90f74d48cdcae9999a3827b7bc2d0d9be4c48cded00174e935ae31e-primary.xml.gz fail: repodata/83455b1e51fd16822747a3b76ae4a64f1278c4a12d489e24ceabe2d456b74f4b-filelists.xml.gz fail: repodata/d2640e2967e3b20f744b7ef16f868ad9a2ff0a67e726f7ab4290f3e8a2a52796-other.xml.gz fail: repodata/b4b72efd32dbe62792bfc7e5472f1f49ef394eb194c039a9489d60be0765deb3-primary.sqlite.xz fail: repodata/4c3d736bcffbe0a560c0a4ac5c3c1279e0b7be7a8c2c576488edf40533ce628a-filelists.sqlite.xz fail: repodata/f20886db08b1b5c617e10aa9e8103defe7e4615608b2ef7d307815dcfe387cff-other.sqlite.xz ok: repodata/b31dded4034511935721017022689f916dee4f46902ec03b200ea455b6c26e1c-comps-Everything.x86_64.xml ok: repodata/9734d400fe2c45922f15a3db02c3f372ac76fc478d3b2a54da4ec702bd69c40a-comps-Everything.x86_64.xml.xz fail: repodata/f57bb5c389af2b3854d91cac02c82e1cadfd764a740e584012f0b41e40811ddf-prestodelta.xml.xz fail: repodata/f6504da72d77acc6b31db0ec0077bc580b424d3e5599f4e94bd2eb59dfd9684a-updateinfo.xml.bz2 08:01:06 root:~ # ./check_repomd_xml_locations /var/repo/epel/8/Everything/aarch64/ fail: repodata/4c4f92104a1abdd848086877c187f83c0fae4f63a929e56fd98f10c6af2fe191-primary.xml.gz fail: repodata/a032b1c806230cb2459b560619c336701b1fdfff664ab368c56309c5bf9292fc-filelists.xml.gz fail: repodata/b8f3b279dfac835813fc5cef13ff5d62a22ac7a160f85c0b7a45d9732581015c-other.xml.gz fail: repodata/55e1359cbdce67ee6c617ef97640c40ad6726cf692d74a5e18322f59a034528f-primary.sqlite.xz fail: repodata/c618dc5b68d6311f3cf1e43fdc23e13b7ae35314c367e6e47f8572f9ac3ef688-filelists.sqlite.xz fail: repodata/29755d3376d443154401a0a030d14873c6147c494d503a6eecdfff5dae24644f-other.sqlite.xz ok: repodata/b31dded4034511935721017022689f916dee4f46902ec03b200ea455b6c26e1c-comps-Everything.aarch64.xml ok: repodata/9734d400fe2c45922f15a3db02c3f372ac76fc478d3b2a54da4ec702bd69c40a-comps-Everything.aarch64.xml.xz fail: repodata/f25d45e0aaf63956851c221a41843beb36b17ee4d29bf600b30f3a03ab34bf2a-prestodelta.xml.xz fail: repodata/f6504da72d77acc6b31db0ec0077bc580b424d3e5599f4e94bd2eb59dfd9684a-updateinfo.xml.bz2 08:01:11 root:~ # ./check_repomd_xml_locations /var/repo/epel/8/Modular/x86_64/ ok: repodata/88e92a01042416eb234919510b5166759003cc8ed94cb4ac7405a1f075f925ee-primary.xml.gz ok: repodata/8adfb892d1a5db04f80d15543935169d7a865c761215dd0397a199362a30370d-filelists.xml.gz ok: repodata/7d5fdce5a87c14cb3b418eaee930a606f4ef1e1ae1da18faaabce1fed6974425-other.xml.gz ok: repodata/b1b3a6319d69628902d5b3997fe97e335933d5567f94a49b3692b1226f7c2adc-modules.yaml.gz ok: repodata/7640949c5874b642397576ccaae1ce8530ce34b992a86cffbdf70fc53397473e-updateinfo.xml.xz ok: repodata/4951f9681328d1e75a4b23de8d9aed7cc3956af5926a1d77ca6dbeacb3e3cba5-updateinfo.xml.zck 08:01:22 root:~ # ./check_repomd_xml_locations /var/repo/epel/8/Modular/aarch64/ ok: repodata/899d964a016140b336752c6c4f0a5a206072c6038416e6ff3c2a72f3a2b00109-primary.xml.gz ok: repodata/ec1a22c875d846067fce2c5094428ac17090669d7af70610bb251ac98bd52942-filelists.xml.gz ok: repodata/408679047e18e372bab78d45d954982f952e5d0a7a95367ca3a993e9f20b4f3c-other.xml.gz ok: repodata/e4319317e8437eb1317a05819652f1a3e4a6531355e7e746e2bc3794d4b806cf-modules.yaml.gz ok: repodata/7640949c5874b642397576ccaae1ce8530ce34b992a86cffbdf70fc53397473e-updateinfo.xml.xz ok: repodata/4951f9681328d1e75a4b23de8d9aed7cc3956af5926a1d77ca6dbeacb3e3cba5-updateinfo.xml.zck 08:01:27 root:~ # ./check_repomd_xml_locations /var/repo/epel/9/Everything/x86_64/ fail: repodata/d891fe86c9e0f51c64f8a2e761f6d3786c50229c04d157018a46acabb6bfed05-primary.xml.gz fail: repodata/837ec7955a7d97abddf6cbff12dd8bdea030b4395d1cad75954abedf0531f5e1-filelists.xml.gz fail: repodata/54b6b8728cadb99c12c9e460f3394d7aee44b6c0e3e0686c4dee0aef3451bf1a-other.xml.gz fail: repodata/648243a1dbe7565e61a01706352a7849b01fbb468b149012b1f251f50a333e5b-primary.sqlite.bz2 fail: repodata/e8010c08eed4fff4add1a904dc7da53bd2b5b692f88290b5e3c0c8cb358974e4-filelists.sqlite.bz2 fail: repodata/e13eba2c9afe37644c7dad8e0c33d6a06537db9a4cd71a1b9f13687bafc167f4-other.sqlite.bz2 ok: repodata/4ec0ae1bc2c5baa2d1e390e68ed92033cb05deea86333bd01351e8f73615e404-comps-Everything.x86_64.xml ok: repodata/16c4c7b5f6a2d6c380d355943b22d53d36dfa0cbe7c533814a8af6fcf07a8686-comps-Everything.x86_64.xml.gz fail: repodata/55024951843f51ad7495aab3e8136dcd6e0af1a603a5390507f481273dbddb3b-prestodelta.xml.gz fail: repodata/c6a33fd683305c045a7ff28badaffd331ac1e4c45e7f0cc8e032c22a5acd4df2-updateinfo.xml.bz2 08:01:40 root:~ # ./check_repomd_xml_locations /var/repo/epel/9/Everything/aarch64/ fail: repodata/7d31f507a393368681be212929ffc5c9e86d3bd5dfd49ccaff1babd9a35a5a9a-primary.xml.gz fail: repodata/dee427817ce45a006692097649098e1f00753db2061802bf1a545de98c5df918-filelists.xml.gz fail: repodata/21f6018f24a6763d2d5562a05035cd9d238bb42c08aa4d446b3f94ed6c3d73df-other.xml.gz fail: repodata/39fee08dc3a13c6243ff51c23bdce47e8fb365418aaa5e8123fde568090d1f60-primary.sqlite.bz2 fail: repodata/3f9ab63a80c0c040879681ed94335064a67436cf7ade8d50895c90ad8e7754af-filelists.sqlite.bz2 fail: repodata/3aa6e8f4ffeb112347b47b8173cc25910a78ad68c10358a9f5ca93d07dbeb2ae-other.sqlite.bz2 ok: repodata/4ec0ae1bc2c5baa2d1e390e68ed92033cb05deea86333bd01351e8f73615e404-comps-Everything.aarch64.xml ok: repodata/16c4c7b5f6a2d6c380d355943b22d53d36dfa0cbe7c533814a8af6fcf07a8686-comps-Everything.aarch64.xml.gz fail: repodata/bb7f5cb46cb63fea914043209d90b61906cc06c9856ae83110bd26bd061daad3-prestodelta.xml.gz fail: repodata/c6a33fd683305c045a7ff28badaffd331ac1e4c45e7f0cc8e032c22a5acd4df2-updateinfo.xml.bz2
To validate the check and rsync status outside of EPEL I've run it against the other 12 mirrors we maintain. These are mirrored using the same method/scripting/filesystems/os/etc, and do not have this issue. Examples: 08:08:59 root:~ # ./check_repomd_xml_locations /var/repo/rocky-linux/8/BaseOS/x86_64/os ok: repodata/d023aea410537b1caa0df7c8b7d17bea8145302b59cf7adc3a9b92119a4f9654-primary.xml.gz ok: repodata/e4c6fc6e80303cf9c27c499b913bd7165e1f542fa622096004b180b8e9f1c603-filelists.xml.gz ok: repodata/a69ab1b6236b04d262a29b1c007d37e94ef5e2a6f7e576f6ce501842198bf424-other.xml.gz ok: repodata/b810761f905a608cd074aac3371aa4952f5bd30d2e77d368c28548572e950a72-primary.sqlite.xz ok: repodata/3e5e8447237bf7a5033ca0a8409d9e88ac6b208711713889a682165c578087bc-filelists.sqlite.xz ok: repodata/476f80e14610eec29d247f070f12e86955c92897e66e21565ca471493231c7f9-other.sqlite.xz ok: repodata/dae7e104812099a2f632ea4c5ef2769aca18ca1205abdd2c3ba6d171e319df3d-comps-BaseOS.x86_64.xml ok: repodata/741d3c80487757df624285f4a107f925abfac16115210486760a3920d8724e38-comps-BaseOS.x86_64.xml.xz ok: repodata/09fba4ce8e1dd769dd48d4f9e347119d1b30826e2710ea7fbfe21e91cbdd8523-updateinfo.xml.gz 08:09:02 root:~ # ./check_repomd_xml_locations /var/repo/rocky-linux/9/AppStream/aarch64/os ok: repodata/b0b7187b-86e0-4eaf-bec8-3802bb794f8a-PRIMARY.xml.gz ok: repodata/6871de8c-4eef-46bc-9f24-929877be2d45-FILELISTS.xml.gz ok: repodata/6871de8c-4eef-46bc-9f24-929877be2d45-OTHER.xml.gz ok: repodata/35e6cafd805abc07279320cdb95e0c8e25c75541214de7c04d84426c99ecc12a-primary.sqlite.gz ok: repodata/7a8c0814c9884a628f9df4e16d4e4a0611e377dfb21ba96e32d0a126084d2bc2-filelists.sqlite.gz ok: repodata/eb304fe6e6a780203743447f13d0e39b5fed2b36fc37e85fec512095848949b5-other.sqlite.gz ok: repodata/6871de8c-4eef-46bc-9f24-929877be2d45-GROUPS.xml ok: repodata/6871de8c-4eef-46bc-9f24-929877be2d45-GROUPS.xml.gz ok: repodata/6871de8c-4eef-46bc-9f24-929877be2d45-MODULES.yaml.gz ok: repodata/351f5b1c-967c-4392-8471-bb7aa0006c07-UPDATEINFO.xml.gz 08:14:31 root:~ # ./check_repomd_xml_locations /var/repo/nvidia/public/repo/mlnx_ofed/latest/rhel8/aarch64/ ok: repodata/c4d462655d6a9f742195959d82e97e92fff7be53c3b861b6dae585f1c4d71615-primary.xml.gz ok: repodata/afd7597806478bfa4f6756facd45d6117d464aef4e735ed2472bb4ef6a89ea8b-filelists.xml.gz ok: repodata/f1d5da69cc0ab7cdc7c7e59a349620177a3632ac6dc06f247010cc34542727bd-other.xml.gz ok: repodata/40aaa7ee67185156190c38c105e07a0a643b3790938ebdff28527678d0e2ea91-primary.sqlite.bz2 ok: repodata/e211035c3d60438d7dc5ef0ae2bf24c27792078d54e15dbbbc1459834cf5fda4-filelists.sqlite.bz2 ok: repodata/a3bb180fb5cee3177db6f29018ad5c246261e8b1ecfcf36401811cc7836dbb95-other.sqlite.bz2 08:14:37 root:~ # ./check_repomd_xml_locations /var/repo/auristor/filesystem/repo/v2021.05/8/aarch64/ ok: repodata/141a9197587b852e8419b106a4f79ae709415533d55f84ef8e880b75af366e15-primary.xml.gz ok: repodata/ae264358436d389155debd25f40faa4d85aa72683b7835d28de68b9bb846ae07-filelists.xml.gz ok: repodata/8c220794c7b653351cbe601231ed9ff660a55d63b5164fa8b1a56abe0aa63606-other.xml.gz ok: repodata/abd2c1e77b2bf4cfacfba4464126fc1b2b40114ce1045b40a7265f37040b1e4b-primary.sqlite.bz2 ok: repodata/0fda89927127594c9027e2e5ceabf74713a0256b8dcc01c0453ffb9aead3b41a-filelists.sqlite.bz2 ok: repodata/69faa91b61d27d829a745d4af62ecde859179296b6c01cceab893fb200126790-other.sqlite.bz2
I don't *think* I'm stepping in ongoing changes to the upstream sources... the results here are uniform through 4 successful rsyncs across as many days. But just to check I've gone through the verbose rsync output for the usual indications (permission denied for "Packages/{letter}/.{name}", presence of "Packages/{letter}/.*.rpm.{tmpstring}", etc. None have been reported in the last week.
I think this is due to EPEL no longer doing modularity but am not sure. I have opened https://pagure.io/releng/issue/11470 for Fedora release engineering to look at it as this is more of a IT issue than a bug issue.
Please note that the problem exists for epel/8/Everything/{arch} and epel/9/Everything/{arch} only. There is no problem with epel/8/Modular/{arch}/repodata/repomd.xml.
Also, note that I'm *not* running createrepo, or referring to any locally produced repo metadata. All our EPEL repo metadata is from upstream via rsync.
Understood. I am wrong about the repomd.xml file being from modularity. I went and looked at the upstream mirror you were using and it looks like it is an incomplete mirror for some reason. On https://dl.fedoraproject.org/pub/epel/8/Everything/aarch64/repodata there is a ``` -rw-r--r--. 1 263 263 1756628 2023-06-09 01:44 29755d3376d443154401a0a030d14873c6147c494d503a6eecdfff5dae24644f-other.sqlite.xz -rw-r--r--. 1 263 263 2903975 2023-06-09 01:43 4c4f92104a1abdd848086877c187f83c0fae4f63a929e56fd98f10c6af2fe191-primary.xml.gz -rw-r--r--. 1 263 263 4426556 2023-06-09 01:44 55e1359cbdce67ee6c617ef97640c40ad6726cf692d74a5e18322f59a034528f-primary.sqlite.xz -rw-r--r--. 1 263 263 25584 2023-06-09 01:43 9734d400fe2c45922f15a3db02c3f372ac76fc478d3b2a54da4ec702bd69c40a-comps-Everything.aarch64.xml.xz -rw-r--r--. 1 263 263 12052509 2023-06-09 01:43 a032b1c806230cb2459b560619c336701b1fdfff664ab368c56309c5bf9292fc-filelists.xml.gz -rw-r--r--. 1 263 263 132413 2023-06-09 01:43 b31dded4034511935721017022689f916dee4f46902ec03b200ea455b6c26e1c-comps-Everything.aarch64.xml -rw-r--r--. 1 263 263 1622493 2023-06-09 01:43 b8f3b279dfac835813fc5cef13ff5d62a22ac7a160f85c0b7a45d9732581015c-other.xml.gz -rw-r--r--. 1 263 263 10501984 2023-06-09 01:44 c618dc5b68d6311f3cf1e43fdc23e13b7ae35314c367e6e47f8572f9ac3ef688-filelists.sqlite.xz -rw-r--r--. 1 263 263 776 2023-06-09 01:44 f25d45e0aaf63956851c221a41843beb36b17ee4d29bf600b30f3a03ab34bf2a-prestodelta.xml.xz -rw-r--r--. 1 263 263 1072131 2023-06-09 01:47 f6504da72d77acc6b31db0ec0077bc580b424d3e5599f4e94bd2eb59dfd9684a-updateinfo.xml.bz2 -rw-r--r--. 1 263 263 4848 2023-06-09 01:47 repomd.xml ``` but none of those files are in rsync://mirror.umd.edu/fedora/epel/ except for the updated repomd.xml.
Could you try a `rsync -n -rvlptDSH --exclude="*.~tmp~" --delete-delay --delay-updates rsync://dl.fedoraproject.org/fedora/epel/ /var/repo/epel/ >> "$logfile" 2>&1` I put in a `-n` to not change things but see what would change.
Done, full upstream list, two noteworthy items: minor: testing/8/Everything/s390x/Packages/p/.plasma-workspace-wallpapers-5.24.6-1.el8.noarch.rpm.r5w18X -- orphaned from a borked build? major: a grep of the verbose log for "repodata" shows only the "repodata/" directories themselves (looks like they are all there for everything I'd expect), and these actual metadata files: testing/8/Modular/ppc64le/drpms/os/repodata/1e4079d1ccee120287c4091454fdc22d8dd696d07d276d1d0705aaf5a93b9e59-filelists.sqlite.bz2 testing/8/Modular/ppc64le/drpms/os/repodata/8497f7e78cd216ec0d03a478f149f4794e314f1c2404bde3d3808a6a0a020e16-other.sqlite.bz2 testing/8/Modular/ppc64le/drpms/os/repodata/ad8858cc8267b17abcfdd8236965a877433f54a29bb75cd8dc4569a5725eeb3f-primary.sqlite.bz2 testing/8/Modular/ppc64le/drpms/os/repodata/repomd.xml No other repodata/* files were listed. Upstream server was http://mirror.umd.edu/.
Same grep in the previous log shows the expected repodata/*xml entries being downloaded in a pass ending Fri 09 Jun 2023 06:55:50 AM EDT.