This service will be undergoing maintenance at 00:00 UTC, 2016-08-01. It is expected to last about 1 hours
Bug 875163 - Repositories published in v2 do not include repodata/*comps.xml*
Repositories published in v2 do not include repodata/*comps.xml*
Status: CLOSED CURRENTRELEASE
Product: Pulp
Classification: Community
Component: rpm-support (Show other bugs)
2.0.6
Unspecified Linux
high Severity high
: ---
: ---
Assigned To: Pradeep Kilambi
Preethi Thomas
: Triaged
Depends On:
Blocks: katello-v2-pulp
  Show dependency treegraph
 
Reported: 2012-11-09 11:58 EST by Brad Buckingham
Modified: 2013-01-09 12:08 EST (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-01-09 12:08:16 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Brad Buckingham 2012-11-09 11:58:48 EST
Description of problem:

Published repositories appear to be missing the *comps.xml* files.  As a result, consumers do not have access to package groups.

Version-Release number of selected component (if applicable):
pulp-server-0.0.335-1.fc16.noarch

How reproducible:
always

Steps to Reproduce:
1. create, import, publish a repository
2. look at the content in the published repodata directory
  
Actual results:

*comps.xml* do not exist

Expected results:

*comps.xml* do exist

Additional info:

# files in importer directory
> ls -1 /var/lib/pulp/working/repos/ACME_Corporation-zoo-zoo/importers/yum_importer/ACME_Corporation-zoo-zoo/repodata
0f5e661ae9924dc7d8ee1128efa7ca4d1ce2e769e4a505f222a64921da935511-filelists.sqlite.bz2
1f5c1988656a4739b8769a65e4a227f9fd39bba2f06cf9991911bfd6ca5dfdbf-primary.xml.gz
3ed33999939f82ef1a3bb0fe8c969be36aaf78dddfffe3a74dfdefec846e4b4b-other.xml.gz
497d2c3f2af57f60a8d289027a091e33be58a9e5c473d0ad5e9177280b7e415a-comps.xml
77f5f403f135f75654754ea483f2fcfe5b817eb524817c21f814c824492e31b1-filelists.xml.gz
8f41a44a145285693dc7fd9272beccd586c7c8a5a9ba5a7b79193b2b2568f834-comps.xml.gz
948965dc60f99a111af4cdd7903827d7bc51ce1c20bcbbb6c2e7821e1409f5bf-updateinfo.xml
948965dc60f99a111af4cdd7903827d7bc51ce1c20bcbbb6c2e7821e1409f5bf-updateinfo.xml.gz
a5589f0230af7c7a35cf10d26a79b8f4efc2e924a59b41ac033b25dc81453a42-primary.sqlite.bz2
b40c5d8b89f23481042457a3f837131b3324e52c0664dd908b9ad69d356d6d41-other.sqlite.bz2
repomd.xml

# files in published directory:
> ls -1 /var/lib/pulp/published/https/repos/ACME_Corporation/dev/custom/zoo/zoo/repodata
173f61075a42730ae6aba929711d942f268dc952d35bb7bde2d989d2c93338bd-other.sqlite.bz2
225e7e57befc471c8aeaa41c0d2a987231f146226bbbd9b15f75165e799e5d69-primary.xml.gz
58f5d6d21d53db78f1b4cfd30d3475ac867fcb1a5975b69ebd2c36bb19d7143e-primary.sqlite.bz2
5a464e8f22383628107032aacd5d827403e9fad32ca1b0c994f913cf11204172-filelists.sqlite.bz2
9fe21961aa5dfaf7fe71eb77dc620b1fe7e95c012355495a46906ea8006453f8-other.xml.gz
b1b503eae437a7f6a87b12dda08f6e951f618a55d04f5b5732fd204d39177083-filelists.xml.gz
b66d89e59d6d3fc4435f54dd066eeb3f34b0150852b61f11bc0d59b9fdd97a6e-updateinfo.xml.xz
repomd.xml
Comment 1 Preethi Thomas 2012-11-09 16:28:03 EST
From my tests


[root@preethi ~]# ls -l /var/lib/pulp/published/https/repos/repos/pulp/pulp/demo_repos/zoo/repodata/
total 40
-rw-r--r--. 1 apache apache 6261 Nov  9 16:26 13a0f668e92eff205f199e27f710767462052dd1ec0ac47d2e799af8bd14f44d-primary.sqlite.bz2
-rw-r--r--. 1 apache apache 2602 Nov  9 16:26 1435e9e7111ce66d8daff604a58f49b060e44d05f3397f091f49d9f2cee32803-other.sqlite.bz2
-rw-r--r--. 1 apache apache 1978 Nov  9 16:26 14ff85c82228d5f61678773088a33b8834ebec16192908afcc76684be7c8687b-filelists.xml.gz
-rw-r--r--. 1 apache apache  540 Nov  9 16:26 213e4a1785e2ed8b79cc6d12e82529f2ef2103bafd40f27c0f97cc65fb96cc77-comps.xml.xz
-rw-r--r--. 1 apache apache 3398 Nov  9 16:26 22b9d7ccb88ae1504b299d7c39a5e48052a418f5362cedc96015a569383a8f22-filelists.sqlite.bz2
-rw-r--r--. 1 apache apache 1815 Nov  9 16:26 3e22578ffdb04671aa8bd251a15a113883676453b7b1c69f1245020d4bcc5788-other.xml.gz
-rw-r--r--. 1 apache apache  564 Nov  9 16:26 61849936bcea98d6b8c7b3bda5a4e6078abd7e050084e4f782de149e6021cfe9-updateinfo.xml.xz
-rw-r--r--. 1 apache apache 3778 Nov  9 16:26 8541771802f8bc1f183ecf575491591fda24fd62da6056b79aeec48b9506048e-primary.xml.gz
-rw-r--r--. 1 apache apache 3807 Nov  9 16:26 repomd.xml
[root@preethi ~]# pulp-admin rpm repo content group --repo-id zooConditional Package Names: 
Default:                   True
Default Package Names:     
Description:               
Display Order:             1024
Id:                        birds
Langonly:                  None
Mandatory Package Names:   stork, cockateel, penguin, duck
Name:                      birds
Optional Package Names:    
Repo Id:                   zoo
Translated Description:    
Translated Name:           
User Visible:              True

Conditional Package Names: 
Default:                   True
Default Package Names:     
Description:               
Display Order:             1024
Id:                        mammals
Langonly:                  None
Mandatory Package Names:   camel, dolphin, lion, zebra, mouse, horse, fox,
                           elephant, cheetah, squirrel, kangaroo, chimpanzee,
                           tiger, giraffe, bear, whale, cow, dog, cat, gorilla,
                           wolf, walrus
Name:                      mammals
Optional Package Names:    
Repo Id:                   zoo
Translated Description:    
Translated Name:           
User Visible:              True


[root@preethi ~]# pulp-admin consumer package-group install --name birds --consumer-id preethi
Install task created with id [1811cd68-3354-48b3-ac05-00adb172ef00]

This command may be exited via ctrl+c without affecting the install.

[-]
Install failed

No Group named birds exists

[root@preethi ~]#
Comment 2 Preethi Thomas 2012-11-14 15:14:12 EST
I reproduced it with the beta build as well. Now both package and package group installs are failing for me.

From agent.log

2012-11-14 14:49:45,378 [INFO][worker-0] dispatch() @ dispatcher.py:646 - request: {'classname': 'Content', 'kws': {}, 'args': [[{'unit_key': {'name': 'birds'}, 'type_id': 'package_group'}], {'apply': True, 'reboot': False, 'importkeys': False}], 'method': 'install', 'cntr': None}
2012-11-14 14:49:45,451 [ERROR][worker-0] install() @ dispatcher.py:80 - handler failed
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pulp/agent/lib/dispatcher.py", line 76, in install
    report = handler.install(conduit, units, dict(options))
  File "/usr/lib/pulp/agent/handlers/rpm.py", line 202, in install
    details = grp.install(names)
  File "/usr/lib/python2.7/site-packages/pulp_rpm/handler/rpmtools.py", line 214, in install
    yb.selectGroup(name)
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 3341, in selectGroup
    if not self.comps.has_group(grpid):
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 1026, in <lambda>
    comps = property(fget=lambda self: self._getGroups(),
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 937, in _getGroups
    raise Errors.GroupsError, _('No Groups Available in any repository')
GroupsError: No Groups Available in any repository
2012-11-14 14:49:45,469 [INFO][worker-0] sendreply() @ rmi.py:156 - 3332d6d2-4348-410a-9922-3cfa545d1365 processed in: 114 (ms)
2012-11-14 14:50:04,048 [INFO][worker-0] dispatch() @ dispatcher.py:646 - request: {'classname': 'Content', 'kws': {}, 'args': [[{'unit_key': {'name': 'zebra'}, 'type_id': 'rpm'}], {'apply': True, 'reboot': False, 'importkeys': False}], 'method': 'install', 'cntr': None}
2012-11-14 14:50:04,270 [ERROR][worker-0] install() @ dispatcher.py:80 - handler failed
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pulp/agent/lib/dispatcher.py", line 76, in install
    report = handler.install(conduit, units, dict(options))
  File "/usr/lib/pulp/agent/handlers/rpm.py", line 99, in install
    details = pkg.install(names)
  File "/usr/lib/python2.7/site-packages/pulp_rpm/handler/rpmtools.py", line 124, in install
    yb.install(pattern=info)
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 4157, in install
    raise Errors.InstallError, _('No package(s) available to install')
InstallError: No package(s) available to install
2012-11-14 14:50:04,272 [INFO][worker-0] sendreply() @ rmi.py:156 - 9b3e4041-b587-4728-a73a-182c8bde510a processed in: 237 (ms)
^C
Comment 3 Pradeep Kilambi 2012-11-15 10:46:49 EST
The repo sync is able to import the comps and publish them just fine. So I dont see any issue wrt sync itself. I dont know if there is any issue on install side, here is what I'm seeing wrt sync:

Via Http:
=========

$ pulp-admin rpm repo create --repo-id=testcomps --feed=http://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/pulp_unittest/ --serve-http=true

Successfully created repository [testcomps]

$ pulp-admin rpm repo sync run --repo-id=testcomps
+----------------------------------------------------------------------+
                  Synchronizing Repository [testcomps]
+----------------------------------------------------------------------+

This command may be exited by pressing ctrl+c without affecting the actual
operation on the server.

Downloading metadata...
[/]
... completed

Downloading repository content...
[==================================================] 100%
RPMs:       3/3 items
Delta RPMs: 0/0 items
Tree Files: 3/3 items
Files:      0/0 items
... completed

Importing errata...
[\]
... completed

Importing package groups/categories...
[\]
... completed

Publishing packages...
[==================================================] 100%
Packages: 0/0 items
... completed

Publishing distributions...
[==================================================] 100%
Distributions: 3/3 items
... completed

Generating metadata
[|]
... completed

Publishing repository over HTTPS
[-]
... skipped

Publishing repository over HTTP
[-]
... completed

Distributor Working Dir:
========================
$ ls -l /var/lib/pulp/working/repos/testcomps/distributors/yum_distributor/repodata/
total 68
-rw-r--r--. 1 apache apache  1046 Nov 15 10:24 6ce3d061770f376bc72ec28e077586b6151513f4711625c3a7ff9b99e323f51c-comps.xml.gz
-rw-r--r--. 1 apache apache 34373 Nov 15 10:24 a4736e625b199d1a4e687a692b0b04acaaad728271f98166bad49f41239434ad-updateinfo.xml.gz
-rw-r--r--. 1 apache apache   584 Nov 15 10:24 filelists.sqlite.bz2
-rw-r--r--. 1 apache apache   219 Nov 15 10:24 filelists.xml.gz
-rw-r--r--. 1 apache apache   571 Nov 15 10:24 other.sqlite.bz2
-rw-r--r--. 1 apache apache   214 Nov 15 10:24 other.xml.gz
-rw-r--r--. 1 apache apache  1214 Nov 15 10:24 primary.sqlite.bz2
-rw-r--r--. 1 apache apache   227 Nov 15 10:24 primary.xml.gz
-rw-r--r--. 1 apache apache  3479 Nov 15 10:24 repomd.xml

Published to Http:
=================

$ ls -l /var/lib/pulp/published/http/repos/repos/pulp/pulp/demo_repos/pulp_unittest/repodata/
total 68
-rw-r--r--. 1 apache apache  1046 Nov 15 10:24 6ce3d061770f376bc72ec28e077586b6151513f4711625c3a7ff9b99e323f51c-comps.xml.gz
-rw-r--r--. 1 apache apache 34373 Nov 15 10:24 a4736e625b199d1a4e687a692b0b04acaaad728271f98166bad49f41239434ad-updateinfo.xml.gz
-rw-r--r--. 1 apache apache   584 Nov 15 10:24 filelists.sqlite.bz2
-rw-r--r--. 1 apache apache   219 Nov 15 10:24 filelists.xml.gz
-rw-r--r--. 1 apache apache   571 Nov 15 10:24 other.sqlite.bz2
-rw-r--r--. 1 apache apache   214 Nov 15 10:24 other.xml.gz
-rw-r--r--. 1 apache apache  1214 Nov 15 10:24 primary.sqlite.bz2
-rw-r--r--. 1 apache apache   227 Nov 15 10:24 primary.xml.gz
-rw-r--r--. 1 apache apache  3479 Nov 15 10:24 repomd.xml

Via Https:
==========

$ pulp-admin rpm repo update --repo-id=testcomps --serve-https=true
Repository [testcomps] successfully updated

$ pulp-admin rpm repo sync run --repo-id=testcomps
+----------------------------------------------------------------------+
                  Synchronizing Repository [testcomps]
+----------------------------------------------------------------------+

This command may be exited by pressing ctrl+c without affecting the actual
operation on the server.

Published to Https:
===================
$ ls -l /var/lib/pulp/published/https/repos/repos/pulp/pulp/demo_repos/pulp_unittest/repodata
total 68
-rw-r--r--. 1 apache apache  1046 Nov 15 10:33 6ce3d061770f376bc72ec28e077586b6151513f4711625c3a7ff9b99e323f51c-comps.xml.gz
-rw-r--r--. 1 apache apache 34373 Nov 15 10:33 a4736e625b199d1a4e687a692b0b04acaaad728271f98166bad49f41239434ad-updateinfo.xml.gz
-rw-r--r--. 1 apache apache   584 Nov 15 10:33 filelists.sqlite.bz2
-rw-r--r--. 1 apache apache   219 Nov 15 10:33 filelists.xml.gz
-rw-r--r--. 1 apache apache   571 Nov 15 10:33 other.sqlite.bz2
-rw-r--r--. 1 apache apache   214 Nov 15 10:33 other.xml.gz
-rw-r--r--. 1 apache apache  1214 Nov 15 10:33 primary.sqlite.bz2
-rw-r--r--. 1 apache apache   227 Nov 15 10:33 primary.xml.gz
-rw-r--r--. 1 apache apache  3480 Nov 15 10:33 repomd.xml

Groups in Pulp:
===============

$ pulp-admin rpm repo content group --repo-id=testcomps
Conditional Package Names: 
Default:                   True
Default Package Names:     system-config-date, system-config-boot
Description:               This group is a collection of graphical
                           administration tools for the system, such as for
                           managing user accounts and configuring system
                           hardware.
Display Order:             1024
Id:                        admin-tools
Langonly:                  None
Mandatory Package Names:   
Name:                      Administration Tools
Optional Package Names:    cacti, yumex
Repo Id:                   testcomps
Translated Description:    
  Af: Hierdie groep is 'n versameling grafiese administrasienutsgoed vir die
      rekenaar, byvoorbeeld vir die bestuur van gebruikersrekeninge en die
      opstelling van rekenaarhardeware.
Translated Name:           
  Af: Administrasienutsgoed
  Zu: Amathuluzi Okuphatha
User Visible:              True
Comment 4 Pradeep Kilambi 2012-11-15 13:25:04 EST
So turns out the issue is with how modifyrepo handles comps xml when merging to repodata. It sets the type to the xml name and since el6 doesnt have --mdtype option we cant pass in the type info. The fix for now is to rename the comps.xml to group.xml so the group gets set.

commit 3e3a6a100c63c90d2ac9c04a9191a673090a51f0
Author: Pradeep Kilambi <pkilambi@redhat.com>
Date:   Thu Nov 15 13:19:52 2012 -0500

    875163 - use group as the xml filename when generating comps so modifyrepo uses that as type id which yum expects

$ yum grouplist
Loaded plugins: presto, product-id, pulp-profile-update, refresh-packagekit, rhnplugin, security, subscription-manager
Updating certificate-based repositories.

Setting up Group Process
testcomps/group                                                                                                                                             | 1.0 kB     00:00     
Available Groups:
   Administration Tools
   DNS Name Server
   Haskell
Done
Comment 5 Jeff Ortel 2012-11-20 17:19:51 EST
build: 2.0.6-0.3.beta
Comment 6 Preethi Thomas 2012-11-26 21:48:23 EST
failing
there still seem to be an issue with package group on the latest build



on the client
[root@fedora17-client ~]# pulp-consumer bind --repo-id fedora17
Consumer [f17-client] successfully bound to repository [fedora17]

[root@fedora17-client ~]# vi /etc/yum.repos.d/pulp.repo 
[root@fedora17-client ~]# yum repolist
fedora17                                                 | 3.3 kB     00:00     
testcomps                                                | 3.7 kB     00:00     
fedora17/primary_db                                      | 343 kB     00:00     
repo id                           repo name                               status
fedora17                          fedora17                                500
pulp-v2-builds                    Pulp v2 Beta Builds                      36
testcomps                         testcomps                                 3
repolist: 539
[root@fedora17-client ~]# yum grouplist
fedora17/group                                           | 265 kB     00:00     
Error: No group data available for configured repositories
[root@fedora17-client ~]# 


on the server

[root@preethi ~]# pulp-admin repo list
+----------------------------------------------------------------------+
                              Repositories
+----------------------------------------------------------------------+

Id:                 test
Display Name:       test
Description:        None
Content Unit Count: 0

Id:                 test2
Display Name:       test2
Description:        None
Content Unit Count: 0

Id:                 centos
Display Name:       centos
Description:        None
Content Unit Count: 3708

Id:                 fedora17
Display Name:       fedora17
Description:        None
Content Unit Count: 3584

Id:                 testcomps
Display Name:       testcomps
Description:        None
Content Unit Count: 61


[root@preethi ~]#  pulp-admin rpm repo content group --repo-id=testcomps
Conditional Package Names: 
Default:                   True
Default Package Names:     system-config-date, system-config-boot
Description:               This group is a collection of graphical
                           administration tools for the system, such as for
                           managing user accounts and configuring system
                           hardware.
Display Order:             1024
Id:                        admin-tools
Langonly:                  None
Mandatory Package Names:   
Name:                      Administration Tools
Optional Package Names:    cacti, yumex
Repo Id:                   testcomps
Translated Description:    
  Af: Hierdie groep is 'n versameling grafiese administrasienutsgoed vir die
      rekenaar, byvoorbeeld vir die bestuur van gebruikersrekeninge en die
      opstelling van rekenaarhardeware.
Translated Name:           
  Af: Administrasienutsgoed
  Zu: Amathuluzi Okuphatha
User Visible:              True

Conditional Package Names: 
Default:                   False
Default Package Names:     bind-chroot
Description:               This package group allows you to run a DNS name
                           server (BIND) on the system.
Display Order:             1024
Id:                        dns-server
Langonly:                  None
Mandatory Package Names:   
Name:                      DNS Name Server
Optional Package Names:    bind, dnsperf
Repo Id:                   testcomps
Translated Description:    
Translated Name:           
  Af: DNS-naambediener
User Visible:              True

Conditional Package Names: 
Default:                   False
Default Package Names:     haskell-platform, alex, happy
Description:               Support for developing programs in the Haskell
                           programming language.
Display Order:             1024
Id:                        haskell
Langonly:                  None
Mandatory Package Names:   
Name:                      Haskell
Optional Package Names:    
Repo Id:                   testcomps
Translated Description:    
Translated Name:           
  As: Haskell
User Visible:              True


[root@preethi ~]#
Comment 7 Pradeep Kilambi 2012-11-27 10:12:58 EST
Looks like the client was missing yum-utils and pyliblzma. After installing these, we can list the groups.

[root@fedora17-client ~]# yum grouplist --disablerepo=* --enablerepo=testcomps
Available Groups:
   Administration Tools
   DNS Name Server
   Haskell
Done

Putting back on qa.
Comment 8 Preethi Thomas 2012-11-27 10:32:58 EST
verified
[root@preethi-el6-pulp ~]# rpm -q pulp-server
pulp-server-2.0.6-0.9.beta.noarch
[root@preethi-el6-pulp ~]# 

[root@fedora16-client ~]# yum grouplist
Setting up Group Process
Available Groups:
   Administration Tools
   DNS Name Server
   Haskell
Done
[root@fedora16-client ~]# pulp-consumer status
This consumer is registered to the server [preethi-el6-pulp.usersys.redhat.com]
with the ID [f16-consumer].

[root@fedora16-client ~]# 
[root@preethi-el6-pulp ~]# pulp-admin rpm consumer package-group install run --consumer-id f16-consumer --name Haskell
Install task created with id [84647d47-c884-413b-9345-d18abe73095c]

This command may be exited via ctrl+c without affecting the install.

[-]
Install Succeeded

Packages for groups already installed

[root@preethi-el6-pulp ~]#
Comment 9 Preethi Thomas 2013-01-09 12:08:16 EST
Pulp v2.0 released

Note You need to log in before you can comment on or make changes to this bug.