Hide Forgot
Description of problem: Adding packagegroup to a repo seems to be broken due to manual updating of repomd.xml. If there is no comps info initially you can add groups to repo and it generates the comps info, but once createrepo is run the comps is written in createrepo repo format as <checksum-comps.xml.gz>. After this if you try to add a package group, it adds the group to repodata/comps.xml but not to <checksum-comps.xml> which what the repomd.xml references. The comps update logic does call update_repomd_xml_file method but thats only effective if there isnt already a group or group_gz. $ sudo pulp-admin -u admin -p admin repo create --id=testrp Successfully created repository [ testrp ] $ ls -l /var/lib/pulp/repos/testrp/repodata/ total 28 -rw-r--r-- 1 apache root 1135 Dec 8 14:13 265697b0adeb4f2c18a08071cef8a7ca63ce7718da1f2a77092d1899f51ebd7b-primary.sqlite.bz2 -rw-r--r-- 1 apache root 184 Dec 8 14:13 28847f8f63ee398add98f54e71f547dbc8c797222db6c238c9aeefca6c0ce03f-primary.xml.gz -rw-r--r-- 1 apache root 171 Dec 8 14:13 3001e302d5f54ed95db43d8ca0246a9b98accf51d26b159d9eb3a11f878bf79d-other.xml.gz -rw-r--r-- 1 apache root 591 Dec 8 14:13 48127ccabe37510857e0519ef2a79d9550eaeb359bb0aea6e1a59d6e0edc3ee6-filelists.sqlite.bz2 -rw-r--r-- 1 apache root 570 Dec 8 14:13 53a7f1f2260125b9e68ed203089a12dc57f97741072167f15c681a6ba13aed6a-other.sqlite.bz2 -rw-r--r-- 1 apache root 175 Dec 8 14:13 b4dd3248effadac316b8efb067af8682514d44d13fbbf82be998d99a0bb086fe-filelists.xml.gz -rw-r--r-- 1 apache root 2971 Dec 8 14:13 repomd.xml $ sudo pulp-admin -u admin -p admin packagegroup create --repoid=testrp --id=pkgrp1 --name=pkgrp1 Package group [pkgrp1] created in repository [testrp] $ ls -l /var/lib/pulp/repos/testrp/repodata/ total 32 -rw-r--r-- 1 apache root 1135 Dec 8 14:13 265697b0adeb4f2c18a08071cef8a7ca63ce7718da1f2a77092d1899f51ebd7b-primary.sqlite.bz2 -rw-r--r-- 1 apache root 184 Dec 8 14:13 28847f8f63ee398add98f54e71f547dbc8c797222db6c238c9aeefca6c0ce03f-primary.xml.gz -rw-r--r-- 1 apache root 171 Dec 8 14:13 3001e302d5f54ed95db43d8ca0246a9b98accf51d26b159d9eb3a11f878bf79d-other.xml.gz -rw-r--r-- 1 apache root 591 Dec 8 14:13 48127ccabe37510857e0519ef2a79d9550eaeb359bb0aea6e1a59d6e0edc3ee6-filelists.sqlite.bz2 -rw-r--r-- 1 apache root 570 Dec 8 14:13 53a7f1f2260125b9e68ed203089a12dc57f97741072167f15c681a6ba13aed6a-other.sqlite.bz2 -rw-r--r-- 1 apache root 175 Dec 8 14:13 b4dd3248effadac316b8efb067af8682514d44d13fbbf82be998d99a0bb086fe-filelists.xml.gz -rw-r--r-- 1 apache root 372 Dec 8 14:13 comps.xml -rw-r--r-- 1 apache root 3149 Dec 8 14:13 repomd.xml $ cat /var/lib/pulp/repos/testrp/repodata/repomd.xml |grep group <data type="group"><location href="repodata/comps.xml"/><checksum type="sha256">075d0e214e18c4c7c751c22a2dabfe7961980c8a15b059abc19daae359199ede</checksum><timestamp>1323371630</timestamp></data></repomd> $ cat /var/lib/pulp/repos/testrp/repodata/comps.xml |grep pkgrp1 <id>pkgrp1</id> <name>pkgrp1</name> $ sudo pulp-admin -u admin -p admin repo generate_metadata --id=testrp Metadata generation has been successfully scheduled for repo id [testrp]. Use --status to check the status. $ less /var/lib/pulp/repos/testrp/repodata/f24aa84cc9b5f0525f94d51f1e90215589d6f463949a05dd876b235f1d05bd95-comps.xml.gz |grep pkgrp1 <id>pkgrp1</id> <name>pkgrp1</name> $ sudo pulp-admin -u admin -p admin packagegroup create --repoid=testrp --id=pkgrp2 --name=pkgrp2 Package group [pkgrp2] created in repository [testrp] $ cat /var/lib/pulp/repos/testrp/repodata/comps.xml |grep pkgrp2 <id>pkgrp2</id> <name>pkgrp2</name> $ less /var/lib/pulp/repos/testrp/repodata/f24aa84cc9b5f0525f94d51f1e90215589d6f463949a05dd876b235f1d05bd95-comps.xml.gz |grep pkgrp2 $ $ less /var/lib/pulp/repos/testrp/repodata/repomd.xml |grep comps <location href="repodata/075d0e214e18c4c7c751c22a2dabfe7961980c8a15b059abc19daae359199ede-comps.xml"/> <location href="repodata/f24aa84cc9b5f0525f94d51f1e90215589d6f463949a05dd876b235f1d05bd95-comps.xml.gz"/> So as you can see any subsequent packagegroup adds will go to comps.xml which is not tracked by repomd.xml and hence lost. Not sure why modifyrepo or createrepo -g isnt leveraged instead of manually mucking with repomd.xml
Proposed fix is to see if modifyrepo is able to update existing data, if it is we will see if it works on el5 & el6, then move logic to using that. Fallback is to check for comps_gz when running our manual repomd.xml update
For more background on this issue. 1) Intent is that when createrepo is run it uses -g and passes in comps.xml. comps.xml is the file that Pulp will update with group information. 2) If a packagegroup is being updated, yet rest of metadata is not changing, then we would do a smaller update and manually update repomd.xml. This was done manually because at the time modifyrepo did not operate on existing entries, it only allowed new entries to be added. Still investigating the behavior noted in comment #0. Not able to duplicate this behavior on my development box, running Fedora 14. Below is output from Fedora 14 $ sudo pulp-admin -u admin -p admin repo create --id=testrp Successfully created repository [ testrp ] $ ls -l /var/lib/pulp/repos/testrp/repodata/ total 28 -rw-r--r--. 1 apache apache 581 Jan 4 10:37 filelists.sqlite.bz2 -rw-r--r--. 1 apache apache 175 Jan 4 10:37 filelists.xml.gz -rw-r--r--. 1 apache apache 573 Jan 4 10:37 other.sqlite.bz2 -rw-r--r--. 1 apache apache 171 Jan 4 10:37 other.xml.gz -rw-r--r--. 1 apache apache 1232 Jan 4 10:37 primary.sqlite.bz2 -rw-r--r--. 1 apache apache 184 Jan 4 10:37 primary.xml.gz -rw-r--r--. 1 apache apache 2684 Jan 4 10:37 repomd.xml $ sudo pulp-admin -u admin -p admin packagegroup create --repoid=testrp --id=pkgrp1 --name=pkgrp1 Package group [pkgrp1] created in repository [testrp] $ ls -l /var/lib/pulp/repos/testrp/repodata/ total 32 -rw-r--r--. 1 apache apache 372 Jan 4 10:37 comps.xml -rw-r--r--. 1 apache apache 581 Jan 4 10:37 filelists.sqlite.bz2 -rw-r--r--. 1 apache apache 175 Jan 4 10:37 filelists.xml.gz -rw-r--r--. 1 apache apache 573 Jan 4 10:37 other.sqlite.bz2 -rw-r--r--. 1 apache apache 171 Jan 4 10:37 other.xml.gz -rw-r--r--. 1 apache apache 1232 Jan 4 10:37 primary.sqlite.bz2 -rw-r--r--. 1 apache apache 184 Jan 4 10:37 primary.xml.gz -rw-r--r--. 1 apache apache 2862 Jan 4 10:37 repomd.xml $ cat /var/lib/pulp/repos/testrp/repodata/repomd.xml |grep group <data type="group"><location href="repodata/comps.xml"/><checksum type="sha256">075d0e214e18c4c7c751c22a2dabfe7961980c8a15b059abc19daae359199ede</checksum><timestamp>1325691461</timestamp></data></repomd> $ sudo pulp-admin -u admin -p admin repo generate_metadata --id=testrp Metadata generation has been successfully scheduled for repo id [testrp]. Use --status to check the status. $ sudo pulp-admin -u admin -p admin repo generate_metadata --id=testrp --status Task Id 195bdc00-36ea-11e1-8478-0022157359f6 Status finished Message None Start Time 2012-01-04 10:38:07-05:00 Finish Time 2012-01-04 10:38:08-05:00 $ ls -l /var/lib/pulp/repos/testrp/repodata/ total 36 -rw-r--r--. 1 apache apache 372 Jan 4 10:38 comps.xml -rw-r--r--. 1 apache apache 296 Jan 4 10:38 comps.xml.gz -rw-r--r--. 1 apache apache 581 Jan 4 10:38 filelists.sqlite.bz2 -rw-r--r--. 1 apache apache 175 Jan 4 10:38 filelists.xml.gz -rw-r--r--. 1 apache apache 573 Jan 4 10:38 other.sqlite.bz2 -rw-r--r--. 1 apache apache 171 Jan 4 10:38 other.xml.gz -rw-r--r--. 1 apache apache 1232 Jan 4 10:38 primary.sqlite.bz2 -rw-r--r--. 1 apache apache 184 Jan 4 10:38 primary.xml.gz -rw-r--r--. 1 apache apache 3236 Jan 4 10:38 repomd.xml
Recap: Issue was present on Fedora-16 only, involves new behavior with createrepo. Essentially issue is that createrepo would take in as input "comps.xml" and rename to "SHA256-comps.xml". In Pulp we never updated the group "location" so Pulp would break when using the new createrepo. What Pulp does: Pulp stores comps info in the mongo DB and in comps.xml. When generating new metadata Pulp will pass in comps.xml to createrepo using the "-g" flag. This is standard behavior and has been working, even with the new createrepo. The broken piece is what happens when Pulp updates a single packagegroup and needs to have this change reflected by repomd.xml. The tool modifyrepo did not support ability to remove/update metadata in el5/el6 or fedora-14 so we wrote our own manual step of updating the XML in repomd.xml for group and group_gz metadata. As of today I checked latest versions of modifyrepo in el5 & el6 and I did not see the remove/update logic present in modifyrepo, so I continued with our Pulp approach of manually updating the XML. The manual step allows a packagegroup update to be completed relatively quickly without waiting for a "createrepo -g comps.xml --update" which would taken a considerably longer period of time to run than just the XML adjustment to repomd.xml Commit http://git.fedorahosted.org/git/?p=pulp.git;a=commitdiff;h=02f6457c908c8fca692c54234194100addeacf42 QE: Follow the test steps Pradeep has outlined in comment #0 The heart of what needs to be tested is. 1) Create empty repo 2) Add a package group 3) Regenerate metadata <- ** Important step 3) Add a 2nd package group 4) Register a yum client to the repo 5) See if the 2nd package group is available. Additionally we have a unit test that covers this behavior in test_comps.py
build: 0.257
verified [root@preethi ~]# pulp-admin -u admin -p admin repo create --id=testrp Successfully created repository [ testrp ] [root@preethi ~]# ls -l /var/lib/pulp/repos/testrp/repodata/ total 28 -rw-r--r--. 1 apache apache 1135 Jan 11 13:18 265697b0adeb4f2c18a08071cef8a7ca63ce7718da1f2a77092d1899f51ebd7b-primary.sqlite.bz2 -rw-r--r--. 1 apache apache 184 Jan 11 13:18 28847f8f63ee398add98f54e71f547dbc8c797222db6c238c9aeefca6c0ce03f-primary.xml.gz -rw-r--r--. 1 apache apache 171 Jan 11 13:18 3001e302d5f54ed95db43d8ca0246a9b98accf51d26b159d9eb3a11f878bf79d-other.xml.gz -rw-r--r--. 1 apache apache 591 Jan 11 13:18 48127ccabe37510857e0519ef2a79d9550eaeb359bb0aea6e1a59d6e0edc3ee6-filelists.sqlite.bz2 -rw-r--r--. 1 apache apache 570 Jan 11 13:18 53a7f1f2260125b9e68ed203089a12dc57f97741072167f15c681a6ba13aed6a-other.sqlite.bz2 -rw-r--r--. 1 apache apache 175 Jan 11 13:18 b4dd3248effadac316b8efb067af8682514d44d13fbbf82be998d99a0bb086fe-filelists.xml.gz -rw-r--r--. 1 apache apache 2971 Jan 11 13:18 repomd.xml [root@preethi ~]# pulp-admin -u admin -p admin packagegroup create --repoid=testrp --id=pkgrp1 --name=pkgrp1 Package group [pkgrp1] created in repository [testrp] [root@preethi ~]# ls -l /var/lib/pulp/repos/testrp/repodata/total 40 -rw-r--r--. 1 apache apache 372 Jan 11 13:21 075d0e214e18c4c7c751c22a2dabfe7961980c8a15b059abc19daae359199ede-comps.xml -rw-r--r--. 1 apache apache 1135 Jan 11 13:18 265697b0adeb4f2c18a08071cef8a7ca63ce7718da1f2a77092d1899f51ebd7b-primary.sqlite.bz2 -rw-r--r--. 1 apache apache 184 Jan 11 13:18 28847f8f63ee398add98f54e71f547dbc8c797222db6c238c9aeefca6c0ce03f-primary.xml.gz -rw-r--r--. 1 apache apache 171 Jan 11 13:18 3001e302d5f54ed95db43d8ca0246a9b98accf51d26b159d9eb3a11f878bf79d-other.xml.gz -rw-r--r--. 1 apache apache 591 Jan 11 13:18 48127ccabe37510857e0519ef2a79d9550eaeb359bb0aea6e1a59d6e0edc3ee6-filelists.sqlite.bz2 -rw-r--r--. 1 apache apache 570 Jan 11 13:18 53a7f1f2260125b9e68ed203089a12dc57f97741072167f15c681a6ba13aed6a-other.sqlite.bz2 -rw-r--r--. 1 apache apache 258 Jan 11 13:21 59167c31a3a7cdeb6c364acba4ed609c09fa0e3961be75c11eb70e1df21158f3-comps.xml.gz -rw-r--r--. 1 apache apache 175 Jan 11 13:18 b4dd3248effadac316b8efb067af8682514d44d13fbbf82be998d99a0bb086fe-filelists.xml.gz -rw-r--r--. 1 apache apache 372 Jan 11 13:21 comps.xml -rw-r--r--. 1 apache apache 3575 Jan 11 13:21 repomd.xml [root@preethi ~]# cat /var/lib/pulp/repos/testrp/repodata/repomd.xml |grep group <data type="group"><location href="repodata/075d0e214e18c4c7c751c22a2dabfe7961980c8a15b059abc19daae359199ede-comps.xml"/><checksum type="sha256">075d0e214e18c4c7c751c22a2dabfe7961980c8a15b059abc19daae359199ede</checksum><timestamp>1326306070</timestamp></data><data type="group_gz"><location href="repodata/59167c31a3a7cdeb6c364acba4ed609c09fa0e3961be75c11eb70e1df21158f3-comps.xml.gz"/><checksum type="sha256">59167c31a3a7cdeb6c364acba4ed609c09fa0e3961be75c11eb70e1df21158f3</checksum><timestamp>1326306070</timestamp><open-checksum>075d0e214e18c4c7c751c22a2dabfe7961980c8a15b059abc19daae359199ede</open-checksum></data></repomd> [root@preethi ~]# [root@preethi ~]# [root@preethi ~]# pulp-admin -u admin -p admin repo generate_metadata --id=testrp Metadata generation has been successfully scheduled for repo id [testrp]. Use --status to check the status. [root@preethi ~]# less /var/lib/pulp/ cache/ init.flag plugins/ repos/ distributions/ packages/ published/ sn.dat [root@preethi ~]# less /var/lib/pulp/repos/testrp/repodata/ 075d0e214e18c4c7c751c22a2dabfe7961980c8a15b059abc19daae359199ede-comps.xml 265697b0adeb4f2c18a08071cef8a7ca63ce7718da1f2a77092d1899f51ebd7b-primary.sqlite.bz2 28847f8f63ee398add98f54e71f547dbc8c797222db6c238c9aeefca6c0ce03f-primary.xml.gz 3001e302d5f54ed95db43d8ca0246a9b98accf51d26b159d9eb3a11f878bf79d-other.xml.gz 48127ccabe37510857e0519ef2a79d9550eaeb359bb0aea6e1a59d6e0edc3ee6-filelists.sqlite.bz2 53a7f1f2260125b9e68ed203089a12dc57f97741072167f15c681a6ba13aed6a-other.sqlite.bz2 59167c31a3a7cdeb6c364acba4ed609c09fa0e3961be75c11eb70e1df21158f3-comps.xml.gz b4dd3248effadac316b8efb067af8682514d44d13fbbf82be998d99a0bb086fe-filelists.xml.gz f24aa84cc9b5f0525f94d51f1e90215589d6f463949a05dd876b235f1d05bd95-comps.xml.gz repomd.xml [root@preethi ~]# less /var/lib/pulp/repos/testrp/repodata/f24aa84cc9b5f0525f94d51f1e90215589d6f463949a05dd876b235f1d05bd95-comps.xml.gz |grep pkggrp1 [root@preethi ~]# [root@preethi ~]# pulp-admin -u admin -p admin packagegroup create --repoid=testrp --id=pkgrp2 --name=pkgrp2 Package group [pkgrp2] created in repository [testrp] [root@preethi ~]# pulp-admin -u admin -p admin repo generate_metadata --id=testrp --status Task Id 8f48127a-3c81-11e1-bce5-002564a85a58 Status finished Message None Start Time 2012-01-11 13:24:55-05:00 Finish Time 2012-01-11 13:24:56-05:00 [root@preethi ~]# ls -l /var/lib/pulp/repos/testrp/repodata/total 44 -rw-r--r--. 1 apache apache 272 Jan 11 13:29 0f644423fde36fe7bf25c2057e188bdf5d464599e2e6d17c45272a776df2e081-comps.xml.gz -rw-r--r--. 1 apache apache 1135 Jan 11 13:24 265697b0adeb4f2c18a08071cef8a7ca63ce7718da1f2a77092d1899f51ebd7b-primary.sqlite.bz2 -rw-r--r--. 1 apache apache 184 Jan 11 13:24 28847f8f63ee398add98f54e71f547dbc8c797222db6c238c9aeefca6c0ce03f-primary.xml.gz -rw-r--r--. 1 apache apache 171 Jan 11 13:24 3001e302d5f54ed95db43d8ca0246a9b98accf51d26b159d9eb3a11f878bf79d-other.xml.gz -rw-r--r--. 1 apache apache 591 Jan 11 13:24 48127ccabe37510857e0519ef2a79d9550eaeb359bb0aea6e1a59d6e0edc3ee6-filelists.sqlite.bz2 -rw-r--r--. 1 apache apache 570 Jan 11 13:24 53a7f1f2260125b9e68ed203089a12dc57f97741072167f15c681a6ba13aed6a-other.sqlite.bz2 -rw-r--r--. 1 apache apache 258 Jan 11 13:21 59167c31a3a7cdeb6c364acba4ed609c09fa0e3961be75c11eb70e1df21158f3-comps.xml.gz -rw-r--r--. 1 apache apache 175 Jan 11 13:24 b4dd3248effadac316b8efb067af8682514d44d13fbbf82be998d99a0bb086fe-filelists.xml.gz -rw-r--r--. 1 apache apache 612 Jan 11 13:29 b8088201f6f1e0b895b94d1905042477581c1b96562c6522795a4b1bbd7dddad-comps.xml -rw-r--r--. 1 apache apache 612 Jan 11 13:29 comps.xml -rw-r--r--. 1 apache apache 3652 Jan 11 13:29 repomd.xml [root@preethi ~]# pulp-admin packagegroup list Usage: pulp-admin <options> packagegroup list <options> pulp-admin: error: Option --repoid is required; please see --help [root@preethi ~]# pulp-admin packagegroup list --repoid=testrp +------------------------------------------+ Repository: testrp Package Group Information +------------------------------------------+ pkgrp1 pkgrp2
Pulp v1.0 is released Closed Current Release.