Hide Forgot
Description of problem: We noticed this with EPEL, but presumably it could impact other repositories. In this case, some of the the 'updateinfo' and 'comps' metadata files are compressed with bzip2. When importing them, pulp doesn't know how to handle the compression so effectively ignores them which just results in empty template files in our imported repos. That doesn't cause a failure or any such, but it means that we don't get errata or comps info for affected repos: Confirmed locally in pulp: <snip> Id: Default_Organization-Red_Hat_Enterprise_Linux_5_EPEL-RHEL5- epel Display Name: RHEL5-epel Description: None Content Unit Counts: Erratum: 1901 Rpm: 7186 <snip> Id: Default_Organization-Red_Hat_Enterprise_Linux_6_EPEL-RHEL6- epel Display Name: RHEL6-epel Description: None Content Unit Counts: Rpm: 13518 </snip> lftp 10.0.1.7:/pulp/repos/Default_Organization/Library/custom/Red_Hat_Enterprise_Linux_6_EPEL/RHEL6-epel/repodata> ls drwxr-xr-x -- .. -rw-r--r-- 223 2016-03-03 04:57 06db691f9df0df2826bfd59e68c3e65fd2e5b41221effc94bb17066bb2577b6f-updateinfo.xml.gz -rw-r--r-- 223 2016-01-01 14:45 09885a9f5781cf6c830af08b6c7c24aaa8bb08c4131e8fbf550fe98c6a598f5e-updateinfo.xml.gz -rw-r--r-- 223 2016-02-24 03:42 2b33ebcf0c6769c680b550c6537a4e06f943fe46ff7551f281c577301650a526-updateinfo.xml.gz -rw-r--r-- 223 2016-02-18 04:30 2f3593eab6167e194f209fa0f2f1b75ca728fa615379e049d5364e94d1b2b3e6-updateinfo.xml.gz -rw-r--r-- 223 2016-03-05 04:49 2f79490636f715d0e9aecf9d38f18de33a728997df336a7266964fa01290b8b7-updateinfo.xml.gz -rw-r--r-- 130 2016-03-09 04:41 3e800f8eb4bfc89f8e320cda3dd709a8e0bef5f4884e3b622df63b17249e8242-comps.xml -rw-r--r-- 2.2M 2016-03-09 04:41 4d5e54195b46b2c2a56c0554b7a8a98b620193b8f9cf4904cf3c35ebf6e0f635-other.xml.gz -rw-r--r-- 223 2016-01-12 13:22 5f72c08dc7ad25826af08ed2adc439e6e6e49c0d4496c7c9eceecb8171fd8a8f-updateinfo.xml.gz -rw-r--r-- 223 2015-12-28 08:06 6f0fe3f26e39119f0f5128c37a7aeddeaba35d52e479fa3cce1316a2f222db7d-updateinfo.xml.gz -rw-r--r-- 223 2016-03-08 05:19 7b93200753c70e42e7a138400e390d10fb76849d3ddebc6b605c7eb5e3ab9020-updateinfo.xml.gz -rw-r--r-- 223 2016-01-15 04:27 9a52a2e142c694025d76ec55239d829ee63f82561d2d6e66168735b1228daa92-updateinfo.xml.gz -rw-r--r-- 223 2016-02-20 04:23 9c147bc018cef2db936cb775421ed7548f08d66ef7e528c3d24cbf90cac886c0-updateinfo.xml.gz -rw-r--r-- 223 2015-12-13 03:52 9cf2bb3f33529b591c433b0b1b0bf957c0fb1f6c0cade66df57a237935e1614b-updateinfo.xml.gz -rw-r--r-- 223 2015-12-12 15:02 9f79321a125f8cb63ace33bf363ea2bae450cf3cc90e9afb521e218e55ae18f7-updateinfo.xml.gz -rw-r--r-- 223 2016-01-14 04:01 20e6b6e562972ec02fba26c045d5badcc3f727c1cb63b4e6db3ecae10fb92e1e-updateinfo.xml.gz -rw-r--r-- 223 2016-01-18 04:31 25fbf871b96152d71bb42b6cfa799958514752700c7050cb9f7e6e2be854933b-updateinfo.xml.gz -rw-r--r-- 223 2016-01-05 10:43 27ddcaba868550a2697a12b15a845c2946660916cf1c8210368aac76090c02ba-updateinfo.xml.gz -rw-r--r-- 223 2016-02-16 04:35 42c114a157ea96be6d3c747481c3b77ad308b4687a30a0a1005548ba8628b30b-updateinfo.xml.gz -rw-r--r-- 223 2016-01-06 12:43 43c6185e831a496a75669b4f222b3bad5e48e23af014be4e2f15d7975fd72448-updateinfo.xml.gz -rw-r--r-- 223 2015-12-20 03:43 60b43cf8ae278687063dc660ab84d31ab3b0d271cd5c70e5a4c579e1908c254a-updateinfo.xml.gz -rw-r--r-- 223 2016-03-04 04:39 64d544c5f1cd5afd8ce5b7902279ed95d2077eeec3602b5c70e84cea71a4e50a-updateinfo.xml.gz -rw-r--r-- 223 2016-03-07 04:59 80d0d5eed68f559ce6feb19ebf1e45b9d6b48fb8f583194f5c5b7ca5587d2887-updateinfo.xml.gz -rw-r--r-- 223 2016-02-12 04:25 82dae105ee468e9c7beaa21617d4bc114e83a09322fb7dbc28e9ac7ff69a55ec-updateinfo.xml.gz -rw-r--r-- 223 2016-01-02 12:13 83eda18c2639a4060fc498bfbd56c44601e0bb80f9980bc2a992878d55cd17cb-updateinfo.xml.gz -rw-r--r-- 223 2016-01-09 13:41 99edc4c05c4a0f0ba945dca46e3df300ac384c15155dc09e6faac0298429f649-updateinfo.xml.gz -rw-r--r-- 223 2016-01-11 14:08 233ba135d1bd4a8dfcea9cbadf55603e5e4ea07c2148944c3bf62f4b5ff4089b-updateinfo.xml.gz -rw-r--r-- 223 2016-01-04 12:50 284de27be429ae8b255036630b40cae4876c0530bd51b0b1e197503db9bf7f81-updateinfo.xml.gz -rw-r--r-- 223 2016-01-08 10:42 337c5ef55c0529b6ddd57ba4c43b40987f6d8f16b638457aa65e5b26f8ad986b-updateinfo.xml.gz -rw-r--r-- 223 2016-03-09 04:41 407f0ee4d91f39be4226bf7e71f35738975934010c6cf44fa7511bccdbe3843f-updateinfo.xml.gz -rw-r--r-- 223 2016-02-19 04:37 574bf0c4ec0ef4c3d646b22b563907de84766922ed2551c3c814df4fac395ff6-updateinfo.xml.gz -rw-r--r-- 223 2016-02-15 04:02 579f09e6ef130005bf2af803761925e528c5b263f20d64a471cfe327e11b7c2b-updateinfo.xml.gz -rw-r--r-- 223 2015-12-21 04:00 604f4eaeba209e7a05ab1fef56e53ebba8c6588260398c07c6cfc29769edc91e-updateinfo.xml.gz -rw-r--r-- 223 2015-12-31 14:07 5167d1e44b04dbfe2552091cad93b2ebabcde2431c5a08b5e949243dc30ce7ce-updateinfo.xml.gz -rw-r--r-- 223 2015-12-15 03:52 8698db10fee9e342a997c4d8d845854210d754c571ad2420b541be96317778e8-updateinfo.xml.gz -rw-r--r-- 223 2016-01-20 04:32 69844ff74abfc497424b08d2e13fc99d052697c3566d48020814c10a516d7d61-updateinfo.xml.gz -rw-r--r-- 223 2016-01-21 04:58 74310d6eb754b7cdf64253d8bbead9b816cbca42a7e0eddb1012bf2d433e7dd5-updateinfo.xml.gz -rw-r--r-- 223 2016-02-28 04:27 92805abcd95a946c4d387288e1a691f7d22abd3d725aed02bb55cfa1089deb86-updateinfo.xml.gz -rw-r--r-- 223 2016-01-27 12:48 105488b53efc07822bd3bd192858d24e7964cfa00ac9c0917ebfb0c25c741420-updateinfo.xml.gz -rw-r--r-- 223 2016-01-17 04:31 117175ade8a20c5548071ca43db8ad7b7a55c13e64cf9513cb0828b0cffa175a-updateinfo.xml.gz -rw-r--r-- 223 2016-01-22 04:29 344229bd6d082cfdc5c930af7ea360885b8905a6fad44e5fb792606f7c0a0bff-updateinfo.xml.gz -rw-r--r-- 223 2016-03-02 04:19 842907e26ca07143ff4e33444141cbf912c213d2649677f26dfc662076db4501-updateinfo.xml.gz -rw-r--r-- 223 2015-12-27 05:29 2500642b55f961c3e2a61bf4d31de01f3418b2a5a423d2d68a3d7c01cb24b929-updateinfo.xml.gz -rw-r--r-- 223 2016-02-17 05:25 7042146d9ab631a135d7e7ccecd4b494bec43769bae16fe1971b143a9337198c-updateinfo.xml.gz -rw-r--r-- 223 2016-02-23 15:53 24328931697646538100e04b6498e120026dd1caa600218cc5e0b5171e2e6724-updateinfo.xml.gz -rw-r--r-- 223 2016-02-22 04:39 a3fe982c96db4226cff709c8cb40d434e1253017afc67a054bbd5da0f98a3f5a-updateinfo.xml.gz -rw-r--r-- 223 2016-03-06 03:57 a4b29a34e3c04a693f129f049ac9034a7625cebc165406029a2be8c1c61b054e-updateinfo.xml.gz -rw-r--r-- 223 2016-01-24 04:32 ada00f814222387ef3ff1ce5214c7769940d3fe7fe6c7edddc9c738cb7540fb0-updateinfo.xml.gz -rw-r--r-- 223 2016-02-25 20:28 b1bd815da66733e341f0e20a730dd04a4179993ccf24ff4137a4f353229d9ae9-updateinfo.xml.gz -rw-r--r-- 223 2015-12-11 12:46 b9f932eea402bf95ee4a71e7af0adfe81a69ba5ba9f94d95a486b0e8b5c0f062-updateinfo.xml.gz -rw-r--r-- 223 2015-12-16 03:49 b295e67f031a6d88740fd9f5ccacb3db84cd3cec92d6e3adad53a5035d6bbdfb-updateinfo.xml.gz -rw-r--r-- 223 2016-02-14 04:26 b784a3988c610e8d070de0c91c15cfca4ec3eda0670bbff21f8f0c32c32cac44-updateinfo.xml.gz -rw-r--r-- 8.1M 2016-03-09 04:41 bf6f872c936d6d81f8a84b4d169efb72a1a98d618678ffee99e53a3eec28bbff-filelists.xml.gz -rw-r--r-- 223 2016-03-01 05:18 bf95ea8b207f65208a1724a1a996a16029dbd193998b05fc63f3298d26516b3c-updateinfo.xml.gz -rw-r--r-- 223 2015-12-17 03:49 c3b0089ca26eecb64780dce95176a9a702665a498ca9bb9f75f094626d659ea7-updateinfo.xml.gz -rw-r--r-- 223 2016-02-11 04:31 c49a77b69a183ad388f4681b74ea1a817529d4a5825b940067e408e6d0efcb86-updateinfo.xml.gz -rw-r--r-- 223 2016-01-03 15:41 cb0298bae69e1cf03a6ecb50f38f717b7c593395eda28648a3c82a4149a7909b-updateinfo.xml.gz -rw-r--r-- 223 2016-02-21 04:31 cc6d8ecad388bf5cae5c5be4a1a6f6cd72c6b257263d55228674ec3869e5ae72-updateinfo.xml.gz -rw-r--r-- 223 2016-02-13 04:32 d4d2d68e405ae7613d9f6d7b26865e423dd2982948af8df454e3583444b98597-updateinfo.xml.gz -rw-r--r-- 223 2016-02-27 04:29 d192a23f251959cf183c65c9cdc041a68d06e88ac7f0bd66ea9bd3b6efe5fd65-updateinfo.xml.gz -rw-r--r-- 223 2016-02-29 04:30 da9770cab091cbd2baf1724c99121b290a38633cc9c2a5bf58fc1753344150b4-updateinfo.xml.gz -rw-r--r-- 223 2016-01-10 12:55 da457275cd8934fd6cb464948cc729c42908bbf891f49c5052ac45071a97f5b4-updateinfo.xml.gz -rw-r--r-- 223 2015-12-19 03:42 da858930f2a612ee4bdfd5ff11b4ad33ebd459cc58aca113d175080a625dfbe0-updateinfo.xml.gz -rw-r--r-- 223 2016-01-19 04:06 dabe08d98474c2091c6dac82a93e824064646c840cfbb4c60124a5c553e1f2dc-updateinfo.xml.gz -rw-r--r-- 223 2016-01-07 13:23 db5fabaa8df0881f2270a7f8f2c1089512cd6b70047a854b939b58b13928cee9-updateinfo.xml.gz -rw-r--r-- 223 2016-01-23 04:27 dcd9cf2f62306298c0ec77f4fe55a89b31b040dd654b7e3ea4d137d22d18a074-updateinfo.xml.gz -rw-r--r-- 223 2015-12-29 13:17 df149dc84c207cb7cba3da963ec5394b3d973353638347f46bb529cf913dd86c-updateinfo.xml.gz -rw-r--r-- 223 2016-01-16 04:19 e32d60ce9c784f39083ea8103ccc45809bbf664234480c88d86b095302f4b35f-updateinfo.xml.gz -rw-r--r-- 223 2015-12-18 03:44 ea7dd3e462dca4956ce9725b9400f6c7ff066b94eb77c980eef509557e2ad90c-updateinfo.xml.gz -rw-r--r-- 223 2015-12-14 03:33 ee54bdcb7148e767f9235e3f6a50ef382cfcd75a7808ded8471d957aabb4957d-updateinfo.xml.gz -rw-r--r-- 223 2015-12-30 12:45 f076e6dff42cf4571766d2319bd74f2f71a2426d83d46adf4d538baed99e9f77-updateinfo.xml.gz -rw-r--r-- 3.4M 2016-03-09 04:41 f4badba745257cc8d214419562cfd067671e8f1a358996a2b08b0f2d23bc3457-primary.xml.gz -rw-r--r-- 223 2015-12-11 11:56 f6e670ba3b153422df4a0fcc9dc27b1474fe2cacb279c8f0c4696be9db1519ee-updateinfo.xml.gz -rw-r--r-- 2.1K 2016-03-09 04:41 repomd.xml </snip> (Obviously we still have the bug where we're not clearing old 'updateinfo' files.. though it might actually stem from a common root issue) Upstream updateinfo (note that it's bz2): <snip> [root@satellite ~]# bzcat dedca5ee69cd418e4dcd76afbc66f64076ad90763f566bd27bd64c7356e3c09d-updateinfo.xml.bz2 | grep '<id>' | wc -l 3893 <snip> Ours from the synced repo: <snip> [root@satellite ~]# zcat 407f0ee4d91f39be4226bf7e71f35738975934010c6cf44fa7511bccdbe3843f-updateinfo.xml.gz | grep '<id>' | wc -l 0 </snip> Upstream: lftp dl.fedoraproject.org:/pub/epel/5/x86_64/repodata> ls drwxr-xr-x -- .. -rw-r--r-- -- 30dbb4059a68020bb80b69d813947ad4e09f843d-filelists.sqlite.bz2 -rw-r--r-- -- 3e42fda7d061ed2ed0f90881e28ab4d7bf2153b0-primary.sqlite.bz2 -rw-r--r-- -- 5c06641838909bc9bdf14054274882c3c9a4c802-comps-el5.xml.bz2 -rw-r--r-- -- 5cfa20dbf25a8d9d59c85275bf323deaee0c18d4-other.xml.gz -rw-r--r-- -- 64ccda8b1668eaafabd95da355d2d794aef3d8cb-filelists.xml.gz -rw-r--r-- -- a39ea8a4fd376b4aa8fcfa3adca38c20837e9987-updateinfo.xml.gz -rw-r--r-- -- f280d8ecba0ce4387370868ed8c2c54d0e4abd9c-primary.xml.gz -rw-r--r-- -- fa8f022647f0ce217f503e4cc7b97ef75d2ed78a-other.sqlite.bz2 -rw-r--r-- -- fe742216803ab0929f98feac1991785f9398aa3a-comps-el5.xml -rw-r--r-- -- repomd.xml lftp dl.fedoraproject.org:/pub/epel/6/x86_64/repodata> ls drwxr-xr-x -- .. -rw-r--r-- -- 13caa651ad4498cafe6e6959349c6469b5a9a30d0ef7a4ca71b2e3b971073d30-other.sqlite.bz2 -rw-r--r-- -- 2a77bce63a6bff832a18fee538c4d29a2ef2e53fa71d7c7d7ff6d6b08ceda99e-filelists.sqlite.bz2 -rw-r--r-- -- 310b15b92caa19e2cd0982a503f65df19de10e5bb16526811bf85dcaea025a84-primary.sqlite.bz2 -rw-r--r-- -- 4f6cd9c63b9b1c69b14ca58e826c6bedf320735a7f2f0182e039f1dd68b85812-comps-el6.xml.bz2 -rw-r--r-- -- 94261f1291cf339f595c289fd0138e98e6389ca6143677bed7be114880b8f0c2-filelists.xml.gz -rw-r--r-- -- b4a2592377857094a7aa98f3f4d5a28ea7487bcca9179c8d29f31168f7da31cd-comps-el6.xml -rw-r--r-- -- c8c3444f570124d03911ca1ff5481e08d839c240a58b95cb84e31da0d98bd155-other.xml.gz -rw-r--r-- -- dedca5ee69cd418e4dcd76afbc66f64076ad90763f566bd27bd64c7356e3c09d-updateinfo.xml.bz2 -rw-r--r-- -- f8bff5cfe70fabb5c31a5e8407d1ec837dfe9ea842b93e1f6f20c7722b888de5-primary.xml.gz -rw-r--r-- -- repomd.xml <snip> Note that the updateinfo and comps are the only two .xml files that are bzipped. Having a look at the other, comps, we see similar: Upstream: <snip> [root@satellite ~]# bzcat 4f6cd9c63b9b1c69b14ca58e826c6bedf320735a7f2f0182e039f1dd68b85812-comps-el6.xml.bz2 | grep '<id>' | wc -l 211 <snip> From our repo: <snip> [root@satellite ~]# cat 3e800f8eb4bfc89f8e320cda3dd709a8e0bef5f4884e3b622df63b17249e8242-comps.xml <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE comps PUBLIC "-//Red Hat, Inc.//DTD Comps info//EN" "comps.dtd"> <comps></comps> [root@satellite ~]# </snip> EPEL5 repo: <snip> lftp dl.fedoraproject.org:/pub/epel/5/x86_64/repodata> ls drwxr-xr-x -- .. -rw-r--r-- -- 30dbb4059a68020bb80b69d813947ad4e09f843d-filelists.sqlite.bz2 -rw-r--r-- -- 3e42fda7d061ed2ed0f90881e28ab4d7bf2153b0-primary.sqlite.bz2 -rw-r--r-- -- 5c06641838909bc9bdf14054274882c3c9a4c802-comps-el5.xml.bz2 -rw-r--r-- -- 5cfa20dbf25a8d9d59c85275bf323deaee0c18d4-other.xml.gz -rw-r--r-- -- 64ccda8b1668eaafabd95da355d2d794aef3d8cb-filelists.xml.gz -rw-r--r-- -- a39ea8a4fd376b4aa8fcfa3adca38c20837e9987-updateinfo.xml.gz -rw-r--r-- -- f280d8ecba0ce4387370868ed8c2c54d0e4abd9c-primary.xml.gz -rw-r--r-- -- fa8f022647f0ce217f503e4cc7b97ef75d2ed78a-other.sqlite.bz2 -rw-r--r-- -- fe742216803ab0929f98feac1991785f9398aa3a-comps-el5.xml -rw-r--r-- -- repomd.xml <snip> RHEL5 pulp repo: <snip> lftp 10.0.1.7:/pulp/repos/Default_Organization/Library/custom/Red_Hat_Enterprise_Linux_5_EPEL/RHEL5-epel/repodata> ls drwxr-xr-x -- .. -rw-r--r-- 485K 2016-02-22 16:24 2e964533f9230d3b572ffcaba7ca3aa774325464-updateinfo.xml.gz -rw-r--r-- 3.6M 2016-02-22 16:24 7f618e6069feb70f7b64929664def3088a2b1904-filelists.xml.gz -rw-r--r-- 130 2016-02-22 16:24 406ca46e811cbb67669659067e496c49a203ebc8-comps.xml -rw-r--r-- 1.6M 2016-02-22 16:24 3305c9a75bcc70c269cf9dfcee1199d92b20c4a9-primary.xml.gz -rw-r--r-- 1.1M 2016-02-22 16:24 ec774d50ba09e3646ba68518b12364a7def23eef-other.xml.gz -rw-r--r-- 1.8K 2016-02-22 16:24 repomd.xml lftp 10.0.1.7:/pulp/repos/Default_Organization/Library/custom/Red_Hat_Enterprise_Linux_5_EPEL/RHEL5-epel/repodata> get 406ca46e811cbb67669659067e496c49a203ebc8-comps.xml 130 bytes transferred lftp 10.0.1.7:/pulp/repos/Default_Organization/Library/custom/Red_Hat_Enterprise_Linux_5_EPEL/RHEL5-epel/repodata> quit [root@satellite ~]# cat 406ca46e811cbb67669659067e496c49a203ebc8-comps.xml <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE comps PUBLIC "-//Red Hat, Inc.//DTD Comps info//EN" "comps.dtd"> <comps></comps> [root@satellite ~]# <snip> I added the following to my local build to add bz2 handling and it appears to have corrected the issue: <patch> [root@satellite repomd]# diff -uNr metadata.py.orig metadata.py --- metadata.py.orig 2016-03-14 16:58:34.157827918 -0400 +++ metadata.py 2016-03-14 17:00:32.741186416 -0400 @@ -4,6 +4,7 @@ from copy import deepcopy import gdbm import gzip +import bz2 import hashlib import logging import lzma @@ -243,6 +244,8 @@ if file_path.endswith('.gz'): file_handle = gzip.open(file_path, 'r') + elif file_path.endswith('.bz2'): + file_handle = bz2.BZ2File(file_path, 'r') elif file_path.endswith('.xz'): file_handle = lzma.LZMAFile(file_path, 'r') else: </patch> <snip> Id: Default_Organization-Red_Hat_Enterprise_Linux_6_EPEL-RHEL6- epel Display Name: RHEL6-epel Description: None Content Unit Counts: Erratum: 3897 Package Category: 3 Package Group: 208 Rpm: 13605 <snip> Confirmation from the updated updateinfo in our local repo: <snip> [fmhirtz@x1 ~]$ zcat b12f20b10fb270270c91a7ba6a7216864022e8b1fdb5c5194abe5d29fb42e6f3-updateinfo.xml.gz | grep '<id>' | wc -l 3897 </snip> Version-Release number of selected component (if applicable): pulp 2.6.0.20-1.el6 How reproducible: Sync EPEL5/6 repo, check errata (and comps). Note lack of errata/comps information in the imported repo.
Created attachment 1136358 [details] Same patch that's in the description
The client confirms that the patch appears to clear up the issues that they were seeing.
Fixed upstream in 2.6.1
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.
Verified in 6.3 snap 5 using Optional RPMs that has some bz2's in repodata.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. > > For information on the advisory, and where to find the updated files, follow the link below. > > If the solution does not work for you, open a new bug report. > > https://access.redhat.com/errata/RHSA-2018:0336