Hide Forgot
Our RHEL-5 hudson instance is not able to sync the Pulp Fedora-16 repos. This may be related to . Below is info on the instance $ cat /etc/redhat-release Red Hat Enterprise Linux Server release 5.7 (Tikanga) $ rpm -q grinder grinder-0.0.136-1.git.0.053ffe1 $ rpm -qa | grep yum yum-security-1.1.16-16.el5 yum-rhn-plugin-0.5.4-22.el5_7.2 yum-3.2.22-37.el5 yum-utils-1.1.16-16.el5 yum-metadata-parser-1.1.2-3.el5 yum-updatesd-0.9-2.el5 $ grinder yum --label test -U http://repos.fedorapeople.org/repos/pulp/pulp/fedora-16/x86_64 grinder.RepoFetch: INFO fetchYumRepo() repo_label = test, repo_url = http://repos.fedorapeople.org/repos/pulp/pulp/fedora-16/x86_64, basepath = ./, verify_options = {} grinder.RepoFetch: INFO test, http://repos.fedorapeople.org/repos/pulp/pulp/fedora-16/x86_64, Calling RepoFetch with: cacert=<None>, clicert=<None>, clikey=<None>, proxy_url=<None>, proxy_port=<3128>, proxy_user=<None>, proxy_pass=<NOT_LOGGED>, sslverify=<1>, max_speed=<None>, verify_options=<{}> grinder.DistroInfo: INFO Preparing to fetch any available trees.. grinder.BaseFetch: INFO Creating directory: ./test grinder.BaseFetch: INFO Fetching Unknown bytes: .treeinfo from http://repos.fedorapeople.org/repos/pulp/pulp/fedora-16/x86_64/.treeinfo grinder.BaseFetch: WARNING Retrying fetch of: .treeinfo with 1 retry attempts left. HTTP status was 404 grinder.BaseFetch: INFO Fetching Unknown bytes: .treeinfo from http://repos.fedorapeople.org/repos/pulp/pulp/fedora-16/x86_64/.treeinfo grinder.BaseFetch: WARNING Retrying fetch of: .treeinfo with 0 retry attempts left. HTTP status was 404 grinder.BaseFetch: INFO Fetching Unknown bytes: .treeinfo from http://repos.fedorapeople.org/repos/pulp/pulp/fedora-16/x86_64/.treeinfo grinder.BaseFetch: WARNING ERROR: Response = 404 fetching http://repos.fedorapeople.org/repos/pulp/pulp/fedora-16/x86_64/.treeinfo. grinder.BaseFetch: INFO Fetching Unknown bytes: treeinfo from http://repos.fedorapeople.org/repos/pulp/pulp/fedora-16/x86_64/treeinfo grinder.BaseFetch: WARNING Retrying fetch of: treeinfo with 1 retry attempts left. HTTP status was 404 grinder.BaseFetch: INFO Fetching Unknown bytes: treeinfo from http://repos.fedorapeople.org/repos/pulp/pulp/fedora-16/x86_64/treeinfo grinder.BaseFetch: WARNING Retrying fetch of: treeinfo with 0 retry attempts left. HTTP status was 404 grinder.BaseFetch: INFO Fetching Unknown bytes: treeinfo from http://repos.fedorapeople.org/repos/pulp/pulp/fedora-16/x86_64/treeinfo grinder.BaseFetch: WARNING ERROR: Response = 404 fetching http://repos.fedorapeople.org/repos/pulp/pulp/fedora-16/x86_64/treeinfo. grinder.DistroInfo: INFO Unable to read the tree info config. grinder.DistroInfo: INFO [Errno 2] No such file or directory: './test/treeinfo' grinder.YumInfo: ERROR Caught exception when trying to fetch content from [http://repos.fedorapeople.org/repos/pulp/pulp/fedora-16/x86_64]: compressed file ended before the logical end-of-stream was detected grinder.ParallelFetch: INFO Grinder stopping grinder.ParallelFetch: INFO stop() invoked grinder.ParallelFetch: INFO stop() completed grinder.ParallelFetch: INFO Told thread <<WorkerThread(Thread-1, initial)>> to stop grinder.ParallelFetch: INFO stop() invoked grinder.ParallelFetch: INFO stop() completed grinder.ParallelFetch: INFO Told thread <<WorkerThread(Thread-2, initial)>> to stop grinder.ParallelFetch: INFO stop() invoked grinder.ParallelFetch: INFO stop() completed grinder.ParallelFetch: INFO Told thread <<WorkerThread(Thread-3, initial)>> to stop grinder.ParallelFetch: INFO stop() invoked grinder.ParallelFetch: INFO stop() completed grinder.ParallelFetch: INFO Told thread <<WorkerThread(Thread-4, initial)>> to stop grinder.ParallelFetch: INFO stop() invoked grinder.ParallelFetch: INFO stop() completed grinder.ParallelFetch: INFO Told thread <<WorkerThread(Thread-5, initial)>> to stop Traceback (most recent call last): File "/usr/bin/grinder", line 23, in ? GrinderCLI.CLI().main() File "/usr/lib/python2.4/site-packages/grinder/GrinderCLI.py", line 433, in main cmd.main() File "/usr/lib/python2.4/site-packages/grinder/GrinderCLI.py", line 58, in main self._do_command() File "/usr/lib/python2.4/site-packages/grinder/GrinderCLI.py", line 271, in _do_command self.yfetch.fetchYumRepo(verify_options=verify_options) File "/usr/lib/python2.4/site-packages/grinder/RepoFetch.py", line 144, in fetchYumRepo self.setupYumInfo() File "/usr/lib/python2.4/site-packages/grinder/RepoFetch.py", line 108, in setupYumInfo info.setUp() File "/usr/lib/python2.4/site-packages/grinder/YumInfo.py", line 343, in setUp skip=self.skip) File "/usr/lib/python2.4/site-packages/grinder/activeobject.py", line 82, in __call__ return self.object(self, *args, **kwargs) File "/usr/lib/python2.4/site-packages/grinder/activeobject.py", line 267, in __call__ return self.__call(method, args, kwargs) File "/usr/lib/python2.4/site-packages/grinder/activeobject.py", line 243, in __call return self.__rmi(method.name, args, kwargs) File "/usr/lib/python2.4/site-packages/grinder/activeobject.py", line 136, in __rmi raise Exception(ex) Exception: Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/grinder/activeobject.py", line 429, in process retval = method(*args, **kwargs) File "/usr/lib/python2.4/site-packages/grinder/YumInfo.py", line 96, in getDownloadItems self.__getRepoData() File "/usr/lib/python2.4/site-packages/grinder/YumInfo.py", line 169, in __getRepoData for ftype in self.__getRepoXmlFileTypes(): File "/usr/lib/python2.4/site-packages/grinder/YumInfo.py", line 156, in __getRepoXmlFileTypes return self.repo.repoXML.fileTypes() File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 1399, in <lambda> repoXML = property(fget=lambda self: self._getRepoXML(), File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 1391, in _getRepoXML self._loadRepoXML(text=self) File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 1381, in _loadRepoXML return self._groupLoadRepoXML(text, ["primary"]) File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 1366, in _groupLoadRepoXML self._commonRetrieveDataMD(mdtypes) File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 1343, in _commonRetrieveDataMD misc.bunzipFile(dl_local, local) File "/usr/lib/python2.4/site-packages/yum/misc.py", line 615, in bunzipFile data = s_fn.read(1024000) EOFError: compressed file ended before the logical end-of-stream was detected
Below is the referenced repomd.xml $ curl http://repos.fedorapeople.org/repos/pulp/pulp/fedora-16/x86_64/repodata/repomd.xml <?xml version="1.0" encoding="UTF-8"?> <repomd xmlns="http://linux.duke.edu/metadata/repo" xmlns:rpm="http://linux.duke.edu/metadata/rpm"> <revision>1323990468</revision> <data type="filelists"> <checksum type="sha256">b1249e06b0d56af5b2b3a7567e8acda6f5d8a5f53bb4e27baa872be035fe3d7d</checksum> <open-checksum type="sha256">3215a91f847c49522b51f699d466383147a74f0f4a6e201e063a6270edc224e6</open-checksum> <location href="repodata/b1249e06b0d56af5b2b3a7567e8acda6f5d8a5f53bb4e27baa872be035fe3d7d-filelists.xml.gz"/> <timestamp>1323990469</timestamp> <size>11743</size> <open-size>172735</open-size> </data> <data type="primary"> <checksum type="sha256">2c8680ed07cdf04274b13bbfc6139ffcc99c518ae7915e1139b9a6a56c4ebff0</checksum> <open-checksum type="sha256">5b63a661fe52f97f87b1bb022e58065b25187409faef8811441f304398708223</open-checksum> <location href="repodata/2c8680ed07cdf04274b13bbfc6139ffcc99c518ae7915e1139b9a6a56c4ebff0-primary.xml.gz"/> <timestamp>1323990469</timestamp> <size>6517</size> <open-size>35608</open-size> </data> <data type="primary_db"> <checksum type="sha256">dc3002c02ccbf2d2d5032a76a77f96a37eb52055b0ad0d091ad5e18185a3b2bf</checksum> <open-checksum type="sha256">25310a54104d243171056a33e5f184e1b009bd9d732730f99efb33d7d2fda46a</open-checksum> <location href="repodata/dc3002c02ccbf2d2d5032a76a77f96a37eb52055b0ad0d091ad5e18185a3b2bf-primary.sqlite.gz"/> <timestamp>1323990469.68</timestamp> <database_version>10</database_version> <size>11451</size> <open-size>51200</open-size> </data> <data type="other_db"> <checksum type="sha256">0e68b8ec6de0d3941c3e3ee90bf6e713129ae4be605f0996d1fdbe725782736d</checksum> <open-checksum type="sha256">5f92825b43db80a70595936c21ea12f252167e4205b4b67045e98cfa8ae3cda9</open-checksum> <location href="repodata/0e68b8ec6de0d3941c3e3ee90bf6e713129ae4be605f0996d1fdbe725782736d-other.sqlite.gz"/> <timestamp>1323990469.45</timestamp> <database_version>10</database_version> <size>15877</size> <open-size>116736</open-size> </data> <data type="other"> <checksum type="sha256">91c7bcc6a1e5b8a9edfb9c8d7992245c6542d374148ab8e8db8f2c473bdc8585</checksum> <open-checksum type="sha256">65e1015a48ea58c25d32fbcdf17df8920d195886dfdbfdfef2d987e3bafbbfb2</open-checksum> <location href="repodata/91c7bcc6a1e5b8a9edfb9c8d7992245c6542d374148ab8e8db8f2c473bdc8585-other.xml.gz"/> <timestamp>1323990469</timestamp> <size>11203</size> <open-size>107228</open-size> </data> <data type="filelists_db"> <checksum type="sha256">bdeea0ce00f937010fb100fbd4757861b4e020d9189d313dd0969f24d60eba0a</checksum> <open-checksum type="sha256">c0af3bea9cd05ac2cc2fcee16d0803b702d56ad1fac6a4b804292c4f5a0fe268</open-checksum> <location href="repodata/bdeea0ce00f937010fb100fbd4757861b4e020d9189d313dd0969f24d60eba0a-filelists.sqlite.gz"/> <timestamp>1323990469.53</timestamp> <database_version>10</database_version> <size>15964</size> <open-size>72704</open-size> </data> </repomd>
Issue looks to be that in Fedora 16 createrepo defaults to compressing the primary.xml file as GZ, yet RHEL-5 clients expect this to be a BZ2 compressed file. Workaround exists on Server when creating repo, use the "--compress-type bz2" option. createrepo --compress-type bz2 . # EXAMPLE with default createrepo on Fedora 16 # ls repodata/ 5273f40a5624f8dcf7204dba94d849d4451482c5dbe8534bb453ed5ebf94840b-filelists.xml.gz 835c33e0f87f5b82a7bbab4f13a43b315a108a0507223399789da094c8a3d5c2-filelists.sqlite.bz2 repomd.xml 572fc7f77a3a084adb66e3e307f8fb90d71dbedc798b0adf2f38493315eb6be2-other.sqlite.bz2 96ff2216324529a8dbb206eb16430c1ddef609bf673bbfb61419acbb0c03f28b-primary.xml.gz 6a132733b73ce734b3c9f672fe20d309afc638321b40a85bfe1b485ec6f72de7-other.xml.gz aa1c2c31c0f1b30baf5b47aa081fa152a204d732e318b1a1ac55ff1f8ececb64-primary.sqlite.bz2 # EXAMPLE with --compress-type bz2 # ls repodata/ 43c45619ca9a768a42e28359a13acacf415cf169c266e7bec7fdaac799106ea5-filelists.sqlite.gz bba480474e4461caf34f5226bf644a39d5e38d4e3e4578d6192c33ee24bb827a-filelists.xml.gz repomd.xml 62a9fba2615302b03fce01939ddc51f16f96e57e5f4b40495542884372f78ad9-other.sqlite.gz f1d676662ff9d91850854c94a1590862dcfdf8d253a3b6cc18ee263b7141ae7c-other.xml.gz b05e7f50dc4c7ffded827bd2d97faf42f46f54e93dd38aee82e88322104ccf04-primary.sqlite.gz f295d996ca64cc9edaf3f32f9fe1579524f32b0db117e79f8db1d0d59968cc17-primary.xml.gz
Note for the previous comment I flipped the example outputs. The issue is with the compression used on the .sqlite files. Older clients expect the .sqlite files if they exist to be compressed with bz2.
Spoke with James and Seth on #yum. Plan is that Seth will change the default behavior of createrepo in Fedora to output metadata compatible for older RHEL clients. This means that *.xml files will be gzipped and *.sqlite files will be bzip2.
QE: Test is 1) Create a simple repo on Fedora-16 with latest createrepo (Refer to bz 771703 to learn what specific version of createrepo is required.) 2) Expose repo through http 3) On a rhel5 client, install grinder and attempt a grinder sync of the repo grinder yum --label test -U http://URL/repo Configuring yum client to the repo and doing a package install is also sufficient.
Moving to ASSIGNED and will wait till bug 771703 is fixed in Fedora 16.
This build has addressed the issue https://admin.fedoraproject.org/updates/createrepo-0.9.9-10.fc16
createrepo-0.9.9-10.fc16 has been pushed to Fedora stable.
verified setup a repo on f16 and on rhel5 did the following tests [root@rhel5-pulp ~]# pulp-admin -u admin -p admin repo create --id=myrepo --feed=http://f16-cds.usersys.redhat.com/myrepo/ Successfully created repository [ myrepo ] [root@rhel5-pulp ~]# pulp-admin -u admin -p admin repo sync --id=myrepo -F Sync for repository myrepo started Sync: Finished 1/1 new items downloaded 0/1 existing items processed Item Details: RPMs: 1/1 [root@rhel5-pulp ~]#
Pulp v1.0 is released Closed Current Release.