Bug 773752

[root@pulp-qe ~]# pulp-admin repo generate_metadata --id=foo
Metadata generation has been successfully scheduled for repo id [foo]. Use --status to check the status.
[root@pulp-qe ~]# pulp-admin repo generate_metadata --status --id=foo

Task Id          	ad7499cc-3d50-11e1-9b04-5254005854de
Status           	error                    
Message          	[Errno 2] No such file or directory
Start Time       	2012-01-12 14:07:31-05:00
Finish Time      	2012-01-12 14:07:31-05:00


[root@pulp-qe ~]# pulp-admin packagegroup install --consumerid=fred --id=bar
Created task id: 26b3d8c7-3d52-11e1-8357-5254005854de
Waiting: [\] 
Consumer ID: fred  [ FAILED ] No Group named bar exists



2012-01-12 14:07:31,521 6753:139821745940224: pulp.server.util:INFO: util:454 Checking what metadata types are available: ['group', 'filelists', 'group_gz', 'primary', 'other', 'filelists_db', 'primary_db', 'other_db']
2012-01-12 14:07:31,523 6753:139821745940224: pulp.server.tasking.task:ERROR: task:455 Task failed: Task ad7499cc-3d50-11e1-9b04-5254005854de: RepoApi._generate_metadata(foo, )
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pulp/server/tasking/task.py", line 404, in run
    result = self.callable(*self.args, **self.kwargs)
  File "/usr/lib/python2.7/site-packages/pulp/server/api/repo.py", line 2229, in _generate_metadata
    pulp.server.util.create_repo(repo_path, checksum_type=repo["checksum_type"])
  File "/usr/lib/python2.7/site-packages/pulp/server/util.py", line 493, in create_repo
    handle = _create_repo(dir, groups=groups, checksum_type=checksum_type)
  File "/usr/lib/python2.7/site-packages/pulp/server/util.py", line 464, in _create_repo
    os.rename(filetype_path, renamed_comps_file)
OSError: [Errno 2] No such file or directory

2012-01-12 14:18:04,276 6753:139822005982976: pulp.server.webservices.controllers.services:INFO: services:361 agent status:   GET received
2012-01-12 14:18:04,442 6753:139821745940224: gofer.rmi.policy:INFO: policy:139 sent (fred):
{
  "classname": "PackageGroups",
  "kws": {},
  "args": [
    [
      "bar"
    ]
  ],
  "method": "install",
  "cntr": null
}
2012-01-12 14:18:05,090 6753:139821745940224: pulp.server.tasking.task:ERROR: task:455 Task failed: Task 26b3d8c7-3d52-11e1-8357-5254005854de: ConsumerApi.__installpackagegroups(fred, ['bar'], )
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pulp/server/tasking/task.py", line 404, in run
    result = self.callable(*self.args, **self.kwargs)
  File "/usr/lib/python2.7/site-packages/pulp/server/api/consumer.py", line 552, in __installpackagegroups
    return pkgrps.install(grpids)
  File "/usr/lib/python2.7/site-packages/gofer/rmi/stub.py", line 72, in __call__
    return self.stub._send(request, opts)
  File "/usr/lib/python2.7/site-packages/gofer/rmi/stub.py", line 133, in _send
    return self.__send(request, options)
  File "/usr/lib/python2.7/site-packages/gofer/rmi/stub.py", line 164, in __send
    any=opts.any)
  File "/usr/lib/python2.7/site-packages/gofer/rmi/policy.py", line 144, in send
    return self.__getreply(sn, reader)
  File "/usr/lib/python2.7/site-packages/gofer/rmi/policy.py", line 181, in __getreply
    return self.__onreply(envelope)
  File "/usr/lib/python2.7/site-packages/gofer/rmi/policy.py", line 197, in __onreply
    raise RemoteException.instance(reply)
GroupsError: No Group named bar exists



[root@pulp-qe pulp]# cd /var/lib/pulp/repos/foo/repodata/
[root@pulp-qe repodata]# ls
0d442d0a6f4ceacac14b2d591391beea672bf52ca260f812e7d1144e89f5d847-primary.sqlite.bz2    d14286fd66dd60939b031a19b123a124be975beb3e6a9a563d3413d65be3703e-other.xml.gz
5be349dcb93480108269d7e697cd2a08462e2171c477f7b63f5e73ac2b9455bd-filelists.sqlite.bz2  e45a9f0d3abcec59e64c0056a5e5d8522e0111201e24a3508bb65c3abf9d883a-filelists.xml.gz
8966e0a688053269cfd0be4ce3b6a08bfe410c973fbf25c6ae20f823006ae6d3-other.sqlite.bz2      fb30cab454ecdbe20257f7bfcefe1f257464c27ae88f06a6e6b9bcc4f716fb08-primary.xml.gz
comps.xml                                                                              repomd.xml
[root@pulp-qe repodata]# cat repomd.xml | grep "comps" 
<data type="group"><location href="repodata/e003685c6005c46c12e6d4bd7cf44e58810adad1c4116fdc295ee24f3767d59a-comps.xml"/><checksum type="sha256">e003685c6005c46c12e6d4bd7cf44e58810adad1c4116fdc295ee24f3767d59a</checksum><timestamp>1326395185</timestamp></data><data type="group_gz"><location href="repodata/6c669d35c67ce93d6d094e623d23d9284b91f8b1a26d57f96f687a30a42bc243-comps.xml.gz"/><checksum type="sha256">6c669d35c67ce93d6d094e623d23d9284b91f8b1a26d57f96f687a30a42bc243</checksum><timestamp>1326395185</timestamp><open-checksum>e003685c6005c46c12e6d4bd7cf44e58810adad1c4116fdc295ee24f3767d59a</open-checksum></data></repomd>


looks like the metadata files need to be adjusted to point to an updated comps.xml as well.

possible solutions:

1. check if the package is associated to the repo the packagegroup belongs to before adding it to the group. If not raise a message and not allow the add operation unless the package is associated to the repo.

alternately,

2. add the package to the packagegroup and also associate it to the repo the package group belongs to as part of 'packagegroup add_package' call.

In util.py we have another reference to "comps.xml" instead of using the filename repomd.xml has for 'groups.

As to #1 and #2 from comment #2, I don't think we want to do this.
Yum allows an empty repo to contain a packagegroup which refers to packages not from that repo, this is valid usage.

I think we can fix this bz with:
  Modify util.py:464 to use the actual filename from the repomd and not "comps.xml".

Issue was with running comps_util update_repomd_xml_file when the comps.xml had not changed.  We were deleting the "old" comps metadata, if we had a situation of no actual change to comps.xml we ended up deleting the current CHECKSUM-comps.xml/gz files.

http://git.fedorahosted.org/git/?p=pulp.git;a=commitdiff;h=4d8ba15cb282e3b0e1ee3b968fd9d543bfe67299


This is what I ran to see the issue and resolution.
pulp-admin repo create --id bz773752
pulp-admin packagegroup create --id=bar --name=bar --repoid=bz773752
pulp-admin packagegroup add_package --id=bar --repoid=bz773752 --name=patb
pulp-admin packagegroup add_package --id=bar --repoid=bz773752 --name=parent
pulp-admin packagegroup info --id=bar --repoid=bz773752
pulp-admin repo content --id=bz773752

pulp-admin repo generate_metadata --id=bz773752
sleep 5
pulp-admin repo generate_metadata --status --id=bz773752

build: 0.263

[root@pulp-qe ~]# pulp-admin content upload -v --nosig /tmp/patb-0.1-1.noarch.rpm 
* Starting Content Upload

* Performing Content Uploads to Pulp server
Successfully uploaded [patb-0.1-1.noarch.rpm] to server

* Content Upload complete.

[root@pulp-qe ~]# pulp-admin content list --orphaned
patb-0.1-1.noarch.rpm,729e328414fb4700ee4f1e44b580d1a0b5362f6ddbfe48f965a87781d89e1272

[root@pulp-qe ~]# pulp-admin repo create --id=foo
Successfully created repository [ foo ]

[root@pulp-qe ~]# pulp-admin packagegroup create --id=bar --repoid=foo --name=bar
Package group [bar] created in repository [foo]

[root@pulp-qe ~]# pulp-admin packagegroup add_package --repoid=foo --id=bar -n patb
Following packages added to group [bar] in repository [foo]: 
 ['patb']

[root@pulp-qe ~]# 

[root@pulp-qe ~]# pulp-admin packagegroup install --consumerid=fred --id=bar
Created task id: fb02667d-4d1f-11e1-8296-5254005854de
Waiting: [\] 
Consumer ID: fred  [ SUCCEEDED ]

Nothing to do

[root@pulp-qe ~]# 

[root@pulp-qe ~]# pulp-admin packagegroup add_package --repoid=foo --id=bar -n patb
Following packages added to group [bar] in repository [foo]: 
 ['patb']

[root@pulp-qe ~]# 

[root@pulp-qe ~]# pulp-admin packagegroup install --consumerid=fred --id=bar
Created task id: 0dd4d9cf-4d20-11e1-bdf1-5254005854de
Waiting: [\] 
Consumer ID: fred  [ SUCCEEDED ]

Nothing to do

[root@pulp-qe ~]# rpm -qa | grep "patb"
[root@pulp-qe ~]# 

FAIL_QA

Adding packages after packagegroup is created is still not working.

David,

Please re-examine this BZ.

We do not want a packagegroup to automatically add a package to a repo.  It is legal for a repo to be empty and have a packagegroup referring to a package from a different repo.


What was fixed in comment #4 was metadata generation being broken.  

For your test in comment #6, that is the expected behavior.  The install of the packagegroup should do nothing since the consumer is not registered to a repo containing that package.

John,

I would suggest adding some warning to the user that the package is not in the repo during add_package. But that's probably a separate RFE bug.

For now: bug verified if the package is in the repo:

[root@pulp-qe log]# pulp-admin -u admin -p admin repo create --id=foo 
Successfully created repository [ foo ]

[root@pulp-qe log]# pulp-admin -u admin -p admin content upload --repoid=foo -v /tmp/patb-0.1-1.noarch.rpm 
* Starting Content Upload

* Performing Content Uploads to Pulp server
Successfully uploaded [patb-0.1-1.noarch.rpm] to server

* Performing Repo Associations 
Packages skipped because of filters associated with the repository foo : 0

* Metadata generation has been scheduled for repository [foo] with a task id [30181a78-4e9e-11e1-9823-5254005854de]; use `pulp-admin repo generate_metadata --status` to check the status.

Content association Complete for Repo [foo]: 
 Packages: 
patb-0.1-1.noarch.rpm 
 
 Files: 
None

* Content Upload complete.
[root@pulp-qe log]# pulp-admin -u admin -p admin packagegroup create --id=bar --repoid=foo --name=bar
Package group [bar] created in repository [foo]

[root@pulp-qe log]# pulp-admin -u admin -p admin packagegroup add_package --repoid=foo --id=bar -n patb
Following packages added to group [bar] in repository [foo]: 
 ['patb']

[root@pulp-qe log]# pulp-consumer -u admin -p admin consumer register --id=fred
warning: this client is not known to the pulp server; run 'pulp-consumer consumer register' to register it
Successfully registered consumer [ fred ]

[root@pulp-qe log]#  pulp-consumer consumer bind --repoid=foo
Successfully subscribed consumer [fred] to repo [foo]

[root@pulp-qe log]# rpm -qa | grep "patb"

[root@pulp-qe log]# pulp-admin -u admin -p admin packagegroup install --consumerid=fred --id=bar
Created task id: 91ee8840-4e9e-11e1-9953-5254005854de
Waiting: [\] 
Consumer ID: fred  [ SUCCEEDED ]
=====================================================================
Package                           Arch     Version    Repository
=====================================================================
Installed:
patb                              noarch   0.1        foo

[root@pulp-qe log]# rpm -qa | grep "patb"
patb-0.1-1.noarch

Pulp v1.0 is released
Closed Current Release.