Bug 790198

Summary:	Uploading a new comps.xml didn't make it into the repo
Product:	[Retired] Pulp	Reporter:	Jay Dobies <jason.dobies>
Component:	user-experience	Assignee:	Pradeep Kilambi <pkilambi>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Preethi Thomas <pthomas>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	1.0.0	CC:	tsanders
Target Milestone:	---	Keywords:	Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2012-02-24 20:11:05 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Jay Dobies 2012-02-13 21:36:35 UTC

This isn't trivial to verify, but here goes:

1. I edited a comps.xml file by hand:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE comps PUBLIC "-//Red Hat, Inc.//DTD Comps info//EN" "comps.dtd">
<comps>
        
  <group>
   <id>test-group-1</id>
   <default>true</default>
   <uservisible>true</uservisible>
   <display_order>1024</display_order>
   <name>Test Group 1</name>
   <description>Test Group 1 Description</description>
    <packagelist>
      <packagereq type="default">gofer</packagereq>
    </packagelist>
  </group>
</comps>


2. I imported that file into a repo.

3. I ran `export` on the repo to verify:

[jdob@pulp1 ~]$ pulp-admin packagegroup export --repoid pulp-f16
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE comps PUBLIC "-//Red Hat, Inc.//DTD Comps info//EN" "comps.dtd">
<comps>
        
  <group>
   <id>test-group-1</id>
   <default>true</default>
   <uservisible>true</uservisible>
   <display_order>1024</display_order>
   <name>Test Group 1</name>
   <description>Test Group 1 Description</description>
    <packagelist>
      <packagereq type="default">gofer</packagereq>
    </packagelist>
  </group>
</comps>


4. I looked at repomd.xml on the repo itself to see which comps file is being used:

[root@pulp1 repodata]# grep comps repomd.xml 
<data type="group"><location href="repodata/df5b12651d61b5efede1b8807fd995ce9090bf9be9e371ce0a2be8a7a1c92058-comps.xml"/><checksum type="sha256">df5b12651d61b5efede1b8807fd995ce9090bf9be9e371ce0a2be8a7a1c92058</checksum><timestamp>1329167952</timestamp></data><data type="group_gz"><location href="repodata/524a8b1b45fd8116264d9aff496db244bcca29e3469b5b7959efdd72706d1a85-comps.xml.gz"/><checksum type="sha256">524a8b1b45fd8116264d9aff496db244bcca29e3469b5b7959efdd72706d1a85</checksum><timestamp>1329167952</timestamp><open-checksum>df5b12651d61b5efede1b8807fd995ce9090bf9be9e371ce0a2be8a7a1c92058</open-checksum></data></repomd>

5. I catted the referenced comps.xml files but never saw my version (what was displayed is the original one before I manually edited and imported a new copy):

[root@pulp1 repodata]# cat df5b12651d61b5efede1b8807fd995ce9090bf9be9e371ce0a2be8a7a1c92058-comps.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE comps PUBLIC "-//Red Hat, Inc.//DTD Comps info//EN" "comps.dtd">
<comps>
        
  <group>
   <id>test-group-1</id>
   <default>true</default>
   <uservisible>true</uservisible>
   <display_order>1024</display_order>
   <name>Test Group 1</name>
   <description>Test Group 1 Description</description>
    <packagelist>
      <packagereq type="default">gofer</packagereq>
    </packagelist>
  </group>        
  <category>
   <id>test-category-2</id>
   <display_order>99</display_order>
   <name>Actual category</name>
   <description></description>
    <grouplist>
    </grouplist>
  </category>

</comps>


6. I unzipped and viewed the .gz version listed in repomd.xml and saw the incorrect comps.xml as well:

[root@pulp1 tmp]# gunzip 524a8b1b45fd8116264d9aff496db244bcca29e3469b5b7959efdd72706d1a85-comps.xml.gz 

[root@pulp1 tmp]# cat comps.xml 
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE comps PUBLIC "-//Red Hat, Inc.//DTD Comps info//EN" "comps.dtd">
<comps>
        
  <group>
   <id>test-group-1</id>
   <default>true</default>
   <uservisible>true</uservisible>
   <display_order>1024</display_order>
   <name>Test Group 1</name>
   <description>Test Group 1 Description</description>
    <packagelist>
      <packagereq type="default">gofer</packagereq>
    </packagelist>
  </group>        
  <category>
   <id>test-category-2</id>
   <display_order>99</display_order>
   <name>Actual category</name>
   <description></description>
    <grouplist>
    </grouplist>
  </category>

</comps>

Comment 1 Jay Dobies 2012-02-13 21:44:09 UTC

I ran the same test again, this time using an altered comps.xml with a group with ID "FINDME" to make grepping easier. Same result.

The export command shows the correct comps.xml:

[jdob@pulp1 ~]$ pulp-admin packagegroup export --repoid pulp-f16
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE comps PUBLIC "-//Red Hat, Inc.//DTD Comps info//EN" "comps.dtd">
<comps>
        
  <group>
   <id>FINDME</id>
   <default>true</default>
   <uservisible>true</uservisible>
   <display_order>1024</display_order>
   <name>Test Group 1</name>
   <description>Test Group 1 Description</description>
    <packagelist>
      <packagereq type="default">gofer</packagereq>
    </packagelist>
  </group>
</comps>



In repodata:

[root@pulp1 repodata]# pwd
/var/lib/pulp/repos/repos/pulp/pulp/v1/testing/fedora-16/x86_64/repodata
[root@pulp1 repodata]# grep FINDME *
[root@pulp1 repodata]# 


And as a sanity check to prove I'm not looking in the wrong location, here's the repo list entry for pulp-f16 to show that it should be the same directory I used above in repodata.


[jdob@pulp1 ~]$ pulp-admin repo list
+------------------------------------------+
       List of Available Repositories
+------------------------------------------+

Id                 	pulp-f16                 
Name               	pulp-f16                 
Repo URL           	https://base-f16-64/pulp/repos/repos/pulp/pulp/v1/testing/fedora-16/x86_64/
Feed URL           	http://repos.fedorapeople.org/repos/pulp/pulp/v1/testing/fedora-16/x86_64
<snip>

Comment 2 Jay Dobies 2012-02-13 21:44:58 UTC

This looks pretty similar to: 761653

Comment 3 Pradeep Kilambi 2012-02-14 14:35:48 UTC

I looked at this briefly after talking to jdob. The issue seems to be that when the comps is imported, its imported into the db but not written to filesystem. SO the metadata in the db is up2date but the comps on the filesystem is still old. 

The fix is to write the newly uploaded comps to filesystem after import and run modifyrepo. Should be a two line change.

Comment 5 Pradeep Kilambi 2012-02-15 20:43:23 UTC

commit 7e666c2ae5198a44b5e33b8d6738b5031cd339c6

Comment 6 Jeff Ortel 2012-02-17 17:17:39 UTC

build: 0.267

Comment 7 Jay Dobies 2012-02-21 14:33:19 UTC

There are two places to check to verify:

- The updated comps is stored in the database. That can be checked using the export command.

- The comps.xml file itself needs to be updated (which is the cause of this bug). Look for the repo on disk (/var/lib/pulp/repos) and dig down to the repodata. Unfortunately, because of how yum works it's not as simple as just looking for comps.xml; for some reason old versions are left around. So check in repomd.xml for the name of the comps file it's currently using. Then look at that file directly and make sure it has your changes.

I found it easiest to put something extremely greppable in the updated file and then just grep for it in all of repodata. See comment #1 for an example.

If you're really feeling ambitious you can verify the sqlite comps database has the correct changes as well. That file will also be referenced by repomd.xml. However, I'm not sure of how to interact with it. You might be able to get reasonable confidence that the referenced sqlite comps file has the same timestamp as the XML version.

Comment 8 Preethi Thomas 2012-02-21 15:11:17 UTC

verified
[root@preethi repodata]# rpm -q pulp
pulp-0.0.267-1.fc15.noarch
[root@preethi repodata]# 

[root@preethi ~]# pulp-admin packagegroup import -r pulp --comps=comps.xml 
Successfully imported comps groups and categories into repository [pulp]
[root@preethi ~]# 
[root@preethi ~]# 
[root@preethi ~]# 
[root@preethi ~]# pulp-admin packagegroup export --repoid pulp
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE comps PUBLIC "-//Red Hat, Inc.//DTD Comps info//EN" "comps.dtd">
<comps>
        
  <group>
   <id>test-group-1</id>
   <default>true</default>
   <uservisible>true</uservisible>
   <display_order>1024</display_order>
   <name>Test Group 1</name>
   <description>Test Group 1 Description</description>
    <packagelist>
      <packagereq type="default">gofer</packagereq>
    </packagelist>
  </group>
</comps>


[root@preethi x86_64]# cd repodata/
[root@preethi repodata]# grep comps repomd.xml 
<data type="group"><location href="repodata/2242d8edd0714cc30fbfcb7fcd97db8b6ed6fa937d46bcd0f3441582ac4e3409-comps.xml"/><checksum type="sha256">2242d8edd0714cc30fbfcb7fcd97db8b6ed6fa937d46bcd0f3441582ac4e3409</checksum><timestamp>1329836723</timestamp></data><data type="group_gz"><location href="repodata/582ddd38c7c2cb96cd24b3d1a4c1003897cf4e2c520865c066f3a7022afe929a-comps.xml.gz"/><checksum type="sha256">582ddd38c7c2cb96cd24b3d1a4c1003897cf4e2c520865c066f3a7022afe929a</checksum><timestamp>1329836723</timestamp><open-checksum>2242d8edd0714cc30fbfcb7fcd97db8b6ed6fa937d46bcd0f3441582ac4e3409</open-checksum></data></repomd>
[root@preethi repodata]# cat 2242d8edd0714cc30fbfcb7fcd97db8b6ed6fa937d46bcd0f3441582ac4e3409-comps.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE comps PUBLIC "-//Red Hat, Inc.//DTD Comps info//EN" "comps.dtd">
<comps>
        
  <group>
   <id>test-group-1</id>
   <default>true</default>
   <uservisible>true</uservisible>
   <display_order>1024</display_order>
   <name>Test Group 1</name>
   <description>Test Group 1 Description</description>
    <packagelist>
      <packagereq type="default">gofer</packagereq>
    </packagelist>
  </group>
</comps>


[root@preethi x86_64]# cd repodata/
[root@preethi repodata]# ls
1c03422c0dfc7814d0ff499309258e223503ecf32bfa3ede66e71da82b54d822-other.sqlite.bz2
2242d8edd0714cc30fbfcb7fcd97db8b6ed6fa937d46bcd0f3441582ac4e3409-comps.xml
582ddd38c7c2cb96cd24b3d1a4c1003897cf4e2c520865c066f3a7022afe929a-comps.xml
703fa04ded4d35d0cd0b0072355abdc5bfb03a456e75773705d40594aff2c876-primary.sqlite.bz2
a2fa153e474a95ffd6d72075a5221451d3c1121ff2082e2d98d868ca0a6ae4a0-primary.xml.gz
a774cc1246d83dce7a196ee90acd678caec60d03f7649a5084e8a60d5e6166c3-filelists.sqlite.bz2
comps.xml
df8095f125cee2a1974ffa59326fba5e2bec7b997d153dd96230804f1f5985d8-filelists.xml.gz
f46deb424242ce8408f3d7963dd6ae255ba891ae811ca30002db0e98678b9f95-other.xml.gz
repomd.xml
[root@preethi repodata]# cat comps.xml 
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE comps PUBLIC "-//Red Hat, Inc.//DTD Comps info//EN" "comps.dtd">
<comps>
        
  <group>
   <id>test-group-1</id>
   <default>true</default>
   <uservisible>true</uservisible>
   <display_order>1024</display_order>
   <name>Test Group 1</name>
   <description>Test Group 1 Description</description>
    <packagelist>
      <packagereq type="default">gofer</packagereq>
    </packagelist>
  </group>
</comps>
[
[root@preethi repodata]# cd 
[root@preethi ~]# vi comps.xml 
[root@preethi ~]# pulp-admin packagegroup list 
Usage: pulp-admin <options> packagegroup list <options>

pulp-admin: error: Option --repoid is required; please see --help
[root@preethi ~]# pulp-admin packagegroup list  --repoid=pulp
+------------------------------------------+
              Repository: pulp
          Package Group Information
+------------------------------------------+
	 test-group-1

[root@preethi ~]# vi comps.xml 
[root@preethi ~]# 
[root@preethi ~]# 
[root@preethi ~]# pulp-admin packagegroup import -r pulp --comps=comps.xml 
Successfully imported comps groups and categories into repository [pulp]
[root@preethi ~]# pulp-admin packagegroup import -r pulp --comps=comps.xml Successfully imported comps groups and categories into repository [pulp]
[root@preethi ~]# pulp-admin packagegroup export --repoid pulp<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE comps PUBLIC "-//Red Hat, Inc.//DTD Comps info//EN" "comps.dtd">
<comps>
        
  <group>
   <id>FINDME</id>
   <default>true</default>
   <uservisible>true</uservisible>
   <display_order>1024</display_order>
   <name>Test Group 1</name>
   <description>Test Group 1 Description</description>
    <packagelist>
      <packagereq type="default">gofer</packagereq>
    </packagelist>
  </group>
</comps>


[root@preethi ~]# cd /var/lib/pulp/repos/repos/pulp/pulp/v1/testing/6Server/x86_64/repodata/
[root@preethi repodata]# pwd
/var/lib/pulp/repos/repos/pulp/pulp/v1/testing/6Server/x86_64/repodata
[root@preethi repodata]# grep FINDME
^C
[root@preethi repodata]# grep FINDME *
comps.xml:   <id>FINDME</id>
ea062218964c50d5b9974c01d2c53cce6972a237d26ce029780ba62754b97dc3-comps.xml:   <id>FINDME</id>
[root@preethi repodata]# 

[root@preethi repodata]# pulp-admin packagegroup list  --repoid=pulp
+------------------------------------------+
              Repository: pulp
          Package Group Information
+------------------------------------------+
	 FINDME

Comment 9 Preethi Thomas 2012-02-24 20:11:05 UTC

Pulp v1.0 is released
Closed Current Release.