Bug 875163

From my tests


[root@preethi ~]# ls -l /var/lib/pulp/published/https/repos/repos/pulp/pulp/demo_repos/zoo/repodata/
total 40
-rw-r--r--. 1 apache apache 6261 Nov  9 16:26 13a0f668e92eff205f199e27f710767462052dd1ec0ac47d2e799af8bd14f44d-primary.sqlite.bz2
-rw-r--r--. 1 apache apache 2602 Nov  9 16:26 1435e9e7111ce66d8daff604a58f49b060e44d05f3397f091f49d9f2cee32803-other.sqlite.bz2
-rw-r--r--. 1 apache apache 1978 Nov  9 16:26 14ff85c82228d5f61678773088a33b8834ebec16192908afcc76684be7c8687b-filelists.xml.gz
-rw-r--r--. 1 apache apache  540 Nov  9 16:26 213e4a1785e2ed8b79cc6d12e82529f2ef2103bafd40f27c0f97cc65fb96cc77-comps.xml.xz
-rw-r--r--. 1 apache apache 3398 Nov  9 16:26 22b9d7ccb88ae1504b299d7c39a5e48052a418f5362cedc96015a569383a8f22-filelists.sqlite.bz2
-rw-r--r--. 1 apache apache 1815 Nov  9 16:26 3e22578ffdb04671aa8bd251a15a113883676453b7b1c69f1245020d4bcc5788-other.xml.gz
-rw-r--r--. 1 apache apache  564 Nov  9 16:26 61849936bcea98d6b8c7b3bda5a4e6078abd7e050084e4f782de149e6021cfe9-updateinfo.xml.xz
-rw-r--r--. 1 apache apache 3778 Nov  9 16:26 8541771802f8bc1f183ecf575491591fda24fd62da6056b79aeec48b9506048e-primary.xml.gz
-rw-r--r--. 1 apache apache 3807 Nov  9 16:26 repomd.xml
[root@preethi ~]# pulp-admin rpm repo content group --repo-id zooConditional Package Names: 
Default:                   True
Default Package Names:     
Description:               
Display Order:             1024
Id:                        birds
Langonly:                  None
Mandatory Package Names:   stork, cockateel, penguin, duck
Name:                      birds
Optional Package Names:    
Repo Id:                   zoo
Translated Description:    
Translated Name:           
User Visible:              True

Conditional Package Names: 
Default:                   True
Default Package Names:     
Description:               
Display Order:             1024
Id:                        mammals
Langonly:                  None
Mandatory Package Names:   camel, dolphin, lion, zebra, mouse, horse, fox,
                           elephant, cheetah, squirrel, kangaroo, chimpanzee,
                           tiger, giraffe, bear, whale, cow, dog, cat, gorilla,
                           wolf, walrus
Name:                      mammals
Optional Package Names:    
Repo Id:                   zoo
Translated Description:    
Translated Name:           
User Visible:              True


[root@preethi ~]# pulp-admin consumer package-group install --name birds --consumer-id preethi
Install task created with id [1811cd68-3354-48b3-ac05-00adb172ef00]

This command may be exited via ctrl+c without affecting the install.

[-]
Install failed

No Group named birds exists

[root@preethi ~]#

I reproduced it with the beta build as well. Now both package and package group installs are failing for me.

From agent.log

2012-11-14 14:49:45,378 [INFO][worker-0] dispatch() @ dispatcher.py:646 - request: {'classname': 'Content', 'kws': {}, 'args': [[{'unit_key': {'name': 'birds'}, 'type_id': 'package_group'}], {'apply': True, 'reboot': False, 'importkeys': False}], 'method': 'install', 'cntr': None}
2012-11-14 14:49:45,451 [ERROR][worker-0] install() @ dispatcher.py:80 - handler failed
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pulp/agent/lib/dispatcher.py", line 76, in install
    report = handler.install(conduit, units, dict(options))
  File "/usr/lib/pulp/agent/handlers/rpm.py", line 202, in install
    details = grp.install(names)
  File "/usr/lib/python2.7/site-packages/pulp_rpm/handler/rpmtools.py", line 214, in install
    yb.selectGroup(name)
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 3341, in selectGroup
    if not self.comps.has_group(grpid):
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 1026, in <lambda>
    comps = property(fget=lambda self: self._getGroups(),
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 937, in _getGroups
    raise Errors.GroupsError, _('No Groups Available in any repository')
GroupsError: No Groups Available in any repository
2012-11-14 14:49:45,469 [INFO][worker-0] sendreply() @ rmi.py:156 - 3332d6d2-4348-410a-9922-3cfa545d1365 processed in: 114 (ms)
2012-11-14 14:50:04,048 [INFO][worker-0] dispatch() @ dispatcher.py:646 - request: {'classname': 'Content', 'kws': {}, 'args': [[{'unit_key': {'name': 'zebra'}, 'type_id': 'rpm'}], {'apply': True, 'reboot': False, 'importkeys': False}], 'method': 'install', 'cntr': None}
2012-11-14 14:50:04,270 [ERROR][worker-0] install() @ dispatcher.py:80 - handler failed
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pulp/agent/lib/dispatcher.py", line 76, in install
    report = handler.install(conduit, units, dict(options))
  File "/usr/lib/pulp/agent/handlers/rpm.py", line 99, in install
    details = pkg.install(names)
  File "/usr/lib/python2.7/site-packages/pulp_rpm/handler/rpmtools.py", line 124, in install
    yb.install(pattern=info)
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 4157, in install
    raise Errors.InstallError, _('No package(s) available to install')
InstallError: No package(s) available to install
2012-11-14 14:50:04,272 [INFO][worker-0] sendreply() @ rmi.py:156 - 9b3e4041-b587-4728-a73a-182c8bde510a processed in: 237 (ms)
^C

The repo sync is able to import the comps and publish them just fine. So I dont see any issue wrt sync itself. I dont know if there is any issue on install side, here is what I'm seeing wrt sync:

Via Http:
=========

$ pulp-admin rpm repo create --repo-id=testcomps --feed=http://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/pulp_unittest/ --serve-http=true

Successfully created repository [testcomps]

$ pulp-admin rpm repo sync run --repo-id=testcomps
+----------------------------------------------------------------------+
                  Synchronizing Repository [testcomps]
+----------------------------------------------------------------------+

This command may be exited by pressing ctrl+c without affecting the actual
operation on the server.

Downloading metadata...
[/]
... completed

Downloading repository content...
[==================================================] 100%
RPMs:       3/3 items
Delta RPMs: 0/0 items
Tree Files: 3/3 items
Files:      0/0 items
... completed

Importing errata...
[\]
... completed

Importing package groups/categories...
[\]
... completed

Publishing packages...
[==================================================] 100%
Packages: 0/0 items
... completed

Publishing distributions...
[==================================================] 100%
Distributions: 3/3 items
... completed

Generating metadata
[|]
... completed

Publishing repository over HTTPS
[-]
... skipped

Publishing repository over HTTP
[-]
... completed

Distributor Working Dir:
========================
$ ls -l /var/lib/pulp/working/repos/testcomps/distributors/yum_distributor/repodata/
total 68
-rw-r--r--. 1 apache apache  1046 Nov 15 10:24 6ce3d061770f376bc72ec28e077586b6151513f4711625c3a7ff9b99e323f51c-comps.xml.gz
-rw-r--r--. 1 apache apache 34373 Nov 15 10:24 a4736e625b199d1a4e687a692b0b04acaaad728271f98166bad49f41239434ad-updateinfo.xml.gz
-rw-r--r--. 1 apache apache   584 Nov 15 10:24 filelists.sqlite.bz2
-rw-r--r--. 1 apache apache   219 Nov 15 10:24 filelists.xml.gz
-rw-r--r--. 1 apache apache   571 Nov 15 10:24 other.sqlite.bz2
-rw-r--r--. 1 apache apache   214 Nov 15 10:24 other.xml.gz
-rw-r--r--. 1 apache apache  1214 Nov 15 10:24 primary.sqlite.bz2
-rw-r--r--. 1 apache apache   227 Nov 15 10:24 primary.xml.gz
-rw-r--r--. 1 apache apache  3479 Nov 15 10:24 repomd.xml

Published to Http:
=================

$ ls -l /var/lib/pulp/published/http/repos/repos/pulp/pulp/demo_repos/pulp_unittest/repodata/
total 68
-rw-r--r--. 1 apache apache  1046 Nov 15 10:24 6ce3d061770f376bc72ec28e077586b6151513f4711625c3a7ff9b99e323f51c-comps.xml.gz
-rw-r--r--. 1 apache apache 34373 Nov 15 10:24 a4736e625b199d1a4e687a692b0b04acaaad728271f98166bad49f41239434ad-updateinfo.xml.gz
-rw-r--r--. 1 apache apache   584 Nov 15 10:24 filelists.sqlite.bz2
-rw-r--r--. 1 apache apache   219 Nov 15 10:24 filelists.xml.gz
-rw-r--r--. 1 apache apache   571 Nov 15 10:24 other.sqlite.bz2
-rw-r--r--. 1 apache apache   214 Nov 15 10:24 other.xml.gz
-rw-r--r--. 1 apache apache  1214 Nov 15 10:24 primary.sqlite.bz2
-rw-r--r--. 1 apache apache   227 Nov 15 10:24 primary.xml.gz
-rw-r--r--. 1 apache apache  3479 Nov 15 10:24 repomd.xml

Via Https:
==========

$ pulp-admin rpm repo update --repo-id=testcomps --serve-https=true
Repository [testcomps] successfully updated

$ pulp-admin rpm repo sync run --repo-id=testcomps
+----------------------------------------------------------------------+
                  Synchronizing Repository [testcomps]
+----------------------------------------------------------------------+

This command may be exited by pressing ctrl+c without affecting the actual
operation on the server.

Published to Https:
===================
$ ls -l /var/lib/pulp/published/https/repos/repos/pulp/pulp/demo_repos/pulp_unittest/repodata
total 68
-rw-r--r--. 1 apache apache  1046 Nov 15 10:33 6ce3d061770f376bc72ec28e077586b6151513f4711625c3a7ff9b99e323f51c-comps.xml.gz
-rw-r--r--. 1 apache apache 34373 Nov 15 10:33 a4736e625b199d1a4e687a692b0b04acaaad728271f98166bad49f41239434ad-updateinfo.xml.gz
-rw-r--r--. 1 apache apache   584 Nov 15 10:33 filelists.sqlite.bz2
-rw-r--r--. 1 apache apache   219 Nov 15 10:33 filelists.xml.gz
-rw-r--r--. 1 apache apache   571 Nov 15 10:33 other.sqlite.bz2
-rw-r--r--. 1 apache apache   214 Nov 15 10:33 other.xml.gz
-rw-r--r--. 1 apache apache  1214 Nov 15 10:33 primary.sqlite.bz2
-rw-r--r--. 1 apache apache   227 Nov 15 10:33 primary.xml.gz
-rw-r--r--. 1 apache apache  3480 Nov 15 10:33 repomd.xml

Groups in Pulp:
===============

$ pulp-admin rpm repo content group --repo-id=testcomps
Conditional Package Names: 
Default:                   True
Default Package Names:     system-config-date, system-config-boot
Description:               This group is a collection of graphical
                           administration tools for the system, such as for
                           managing user accounts and configuring system
                           hardware.
Display Order:             1024
Id:                        admin-tools
Langonly:                  None
Mandatory Package Names:   
Name:                      Administration Tools
Optional Package Names:    cacti, yumex
Repo Id:                   testcomps
Translated Description:    
  Af: Hierdie groep is 'n versameling grafiese administrasienutsgoed vir die
      rekenaar, byvoorbeeld vir die bestuur van gebruikersrekeninge en die
      opstelling van rekenaarhardeware.
Translated Name:           
  Af: Administrasienutsgoed
  Zu: Amathuluzi Okuphatha
User Visible:              True

So turns out the issue is with how modifyrepo handles comps xml when merging to repodata. It sets the type to the xml name and since el6 doesnt have --mdtype option we cant pass in the type info. The fix for now is to rename the comps.xml to group.xml so the group gets set.

commit 3e3a6a100c63c90d2ac9c04a9191a673090a51f0
Author: Pradeep Kilambi <pkilambi>
Date:   Thu Nov 15 13:19:52 2012 -0500

    875163 - use group as the xml filename when generating comps so modifyrepo uses that as type id which yum expects

$ yum grouplist
Loaded plugins: presto, product-id, pulp-profile-update, refresh-packagekit, rhnplugin, security, subscription-manager
Updating certificate-based repositories.

Setting up Group Process
testcomps/group                                                                                                                                             | 1.0 kB     00:00     
Available Groups:
   Administration Tools
   DNS Name Server
   Haskell
Done

build: 2.0.6-0.3.beta

failing
there still seem to be an issue with package group on the latest build



on the client
[root@fedora17-client ~]# pulp-consumer bind --repo-id fedora17
Consumer [f17-client] successfully bound to repository [fedora17]

[root@fedora17-client ~]# vi /etc/yum.repos.d/pulp.repo 
[root@fedora17-client ~]# yum repolist
fedora17                                                 | 3.3 kB     00:00     
testcomps                                                | 3.7 kB     00:00     
fedora17/primary_db                                      | 343 kB     00:00     
repo id                           repo name                               status
fedora17                          fedora17                                500
pulp-v2-builds                    Pulp v2 Beta Builds                      36
testcomps                         testcomps                                 3
repolist: 539
[root@fedora17-client ~]# yum grouplist
fedora17/group                                           | 265 kB     00:00     
Error: No group data available for configured repositories
[root@fedora17-client ~]# 


on the server

[root@preethi ~]# pulp-admin repo list
+----------------------------------------------------------------------+
                              Repositories
+----------------------------------------------------------------------+

Id:                 test
Display Name:       test
Description:        None
Content Unit Count: 0

Id:                 test2
Display Name:       test2
Description:        None
Content Unit Count: 0

Id:                 centos
Display Name:       centos
Description:        None
Content Unit Count: 3708

Id:                 fedora17
Display Name:       fedora17
Description:        None
Content Unit Count: 3584

Id:                 testcomps
Display Name:       testcomps
Description:        None
Content Unit Count: 61


[root@preethi ~]#  pulp-admin rpm repo content group --repo-id=testcomps
Conditional Package Names: 
Default:                   True
Default Package Names:     system-config-date, system-config-boot
Description:               This group is a collection of graphical
                           administration tools for the system, such as for
                           managing user accounts and configuring system
                           hardware.
Display Order:             1024
Id:                        admin-tools
Langonly:                  None
Mandatory Package Names:   
Name:                      Administration Tools
Optional Package Names:    cacti, yumex
Repo Id:                   testcomps
Translated Description:    
  Af: Hierdie groep is 'n versameling grafiese administrasienutsgoed vir die
      rekenaar, byvoorbeeld vir die bestuur van gebruikersrekeninge en die
      opstelling van rekenaarhardeware.
Translated Name:           
  Af: Administrasienutsgoed
  Zu: Amathuluzi Okuphatha
User Visible:              True

Conditional Package Names: 
Default:                   False
Default Package Names:     bind-chroot
Description:               This package group allows you to run a DNS name
                           server (BIND) on the system.
Display Order:             1024
Id:                        dns-server
Langonly:                  None
Mandatory Package Names:   
Name:                      DNS Name Server
Optional Package Names:    bind, dnsperf
Repo Id:                   testcomps
Translated Description:    
Translated Name:           
  Af: DNS-naambediener
User Visible:              True

Conditional Package Names: 
Default:                   False
Default Package Names:     haskell-platform, alex, happy
Description:               Support for developing programs in the Haskell
                           programming language.
Display Order:             1024
Id:                        haskell
Langonly:                  None
Mandatory Package Names:   
Name:                      Haskell
Optional Package Names:    
Repo Id:                   testcomps
Translated Description:    
Translated Name:           
  As: Haskell
User Visible:              True


[root@preethi ~]#

Looks like the client was missing yum-utils and pyliblzma. After installing these, we can list the groups.

[root@fedora17-client ~]# yum grouplist --disablerepo=* --enablerepo=testcomps
Available Groups:
   Administration Tools
   DNS Name Server
   Haskell
Done

Putting back on qa.

verified
[root@preethi-el6-pulp ~]# rpm -q pulp-server
pulp-server-2.0.6-0.9.beta.noarch
[root@preethi-el6-pulp ~]# 

[root@fedora16-client ~]# yum grouplist
Setting up Group Process
Available Groups:
   Administration Tools
   DNS Name Server
   Haskell
Done
[root@fedora16-client ~]# pulp-consumer status
This consumer is registered to the server [preethi-el6-pulp.usersys.redhat.com]
with the ID [f16-consumer].

[root@fedora16-client ~]# 
[root@preethi-el6-pulp ~]# pulp-admin rpm consumer package-group install run --consumer-id f16-consumer --name Haskell
Install task created with id [84647d47-c884-413b-9345-d18abe73095c]

This command may be exited via ctrl+c without affecting the install.

[-]
Install Succeeded

Packages for groups already installed

[root@preethi-el6-pulp ~]#

Pulp v2.0 released