Bug 974235

Summary: repo copy with --recursive fails for group copy if you run rpm copy --recursive first
Product: [Retired] Pulp Reporter: Preethi Thomas <pthomas>
Component: user-experienceAssignee: Barnaby Court <bcourt>
Status: CLOSED CURRENTRELEASE QA Contact: Preethi Thomas <pthomas>
Severity: medium Docs Contact:
Priority: medium    
Version: 2.2 BetaCC: bcourt, mhrivnak, rbarlow
Target Milestone: ---Keywords: Triaged
Target Release: 2.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-12-09 14:31:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Preethi Thomas 2013-06-13 18:34:46 UTC
Description of problem:

If you run rpm copy rpm --recursive first on an rpm and then run rpm copy group --recursive on a group that contains the rpm it blows up

Version-Release number of selected component (if applicable):

[root@hp-dl360g6-01 ~]# rpm -q pulp-server
pulp-server-2.2.0-0.3.beta.fc18.noarch
[root@hp-dl360g6-01 ~]# 

How reproducible:


Steps to Reproduce:
[root@hp-dl360g6-01 ~]# pulp-admin rpm repo create --repo-id pulp-copySuccessfully created repository [pulp-copy]

[root@hp-dl360g6-01 ~]# pulp-admin rpm repo copy rpm  --from-repo-id pulp --to-repo-id pulp-copy --str-eq "name=pulp-server" --recursive
This command may be exited via ctrl+c without affecting the request.

[-]
Waiting to begin...

[-]
Running...

Units:
  pulp-server-2.2.0-0.3.beta.fc18-noarch

[root@hp-dl360g6-01 ~]# pulp-admin rpm repo copy group --from-repo-id pulp --to-repo-id pulp-copy --str-eq "id=pulp-server" --recursive 
This command may be exited via ctrl+c without affecting the request.

[-]
Waiting to begin...

[-]
Running...

Task Failed

[u'PulpExecutionException: Pulp exception occurred: PulpExecutionException\n']


Actual results:


Expected results:


Additional info:


2013-06-13 14:33:30,313 pulp.server.webservices.controllers.repositories:INFO: Itinerary: ['c6351969-8da4-49e2-aec1-4b2c5abb1aab']
2013-06-13 14:33:30,389 pulp.server.dispatch.task:INFO: SUCCESS: Task c6351969-8da4-49e2-aec1-4b2c5abb1aab: CallRequest: RepoManager.delete_repo(u'pulp-copy')
2013-06-13 14:33:37,898 pulp.server.dispatch.task:INFO: SUCCESS: Task 57738b5a-5140-49b3-8ab9-88e25c1a8ff3: CallRequest: RepoManager.create_and_configure_repo('pulp-copy', None, None, {'_repo-type': 'rpm-repo'}, importer_repo_plugin_config=****, distributor_list=****, importer_type_id='yum_importer')
2013-06-13 14:33:53,326 pulp.server.dispatch.task:INFO: SUCCESS: Task 27447159-bbcd-4bc8-baf4-9059469cef1d: CallRequest: RepoUnitAssociationManager.associate_from_repo('pulp', u'pulp-copy', import_config_override=****, criteria=****)
2013-06-13 14:34:02,598 pulp.plugins.conduits.mixins:ERROR: Exception from server requesting all content units for repository [pulp-copy]
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pulp/plugins/conduits/mixins.py", line 617, in do_get_repo_units
    u = common_utils.to_plugin_associated_unit(unit, type_defs[type_id])
  File "/usr/lib/python2.7/site-packages/pulp/plugins/conduits/_common.py", line 92, in to_plugin_associated_unit
    unit_key[k] = pulp_unit['metadata'].pop(k)
KeyError: u'epoch'
2013-06-13 14:34:02,599 pulp.server.managers.repo.unit_association:ERROR: Exception from importer [yum_importer] while importing units into repository [pulp-copy]
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/unit_association.py", line 276, in associate_from_repo
    call_config, units=transfer_units)
  File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/importer.py", line 68, in import_units
    return associate.associate(source_repo, dest_repo, import_conduit, config, units)
  File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/associate.py", line 84, in associate
    names_to_copy = get_rpms_to_copy_by_name(rpm_names, import_conduit)
  File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/associate.py", line 141, in get_rpms_to_copy_by_name
    for unit in units:
  File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/existing.py", line 65, in get_existing_units
    for result in search_method(criteria):
  File "/usr/lib/python2.7/site-packages/pulp/plugins/conduits/unit_import.py", line 145, in get_destination_units
    return mixins.do_get_repo_units(self.dest_repo_id, criteria, ImporterConduitException)
  File "/usr/lib/python2.7/site-packages/pulp/plugins/conduits/mixins.py", line 617, in do_get_repo_units
    u = common_utils.to_plugin_associated_unit(unit, type_defs[type_id])
  File "/usr/lib/python2.7/site-packages/pulp/plugins/conduits/_common.py", line 92, in to_plugin_associated_unit
    unit_key[k] = pulp_unit['metadata'].pop(k)
ImporterConduitException: u'epoch'
2013-06-13 14:34:02,599 pulp.server.dispatch.task:ERROR: Pulp exception occurred: PulpExecutionException
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pulp/server/dispatch/task.py", line 138, in _run
    result = call(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/unit_association.py", line 276, in associate_from_repo
    call_config, units=transfer_units)
  File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/importer.py", line 68, in import_units
    return associate.associate(source_repo, dest_repo, import_conduit, config, units)
  File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/associate.py", line 84, in associate
    names_to_copy = get_rpms_to_copy_by_name(rpm_names, import_conduit)
  File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/associate.py", line 141, in get_rpms_to_copy_by_name
    for unit in units:
  File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/existing.py", line 65, in get_existing_units
    for result in search_method(criteria):
  File "/usr/lib/python2.7/site-packages/pulp/plugins/conduits/unit_import.py", line 145, in get_destination_units
    return mixins.do_get_repo_units(self.dest_repo_id, criteria, ImporterConduitException)
  File "/usr/lib/python2.7/site-packages/pulp/plugins/conduits/mixins.py", line 617, in do_get_repo_units
    u = common_utils.to_plugin_associated_unit(unit, type_defs[type_id])
  File "/usr/lib/python2.7/site-packages/pulp/plugins/conduits/_common.py", line 92, in to_plugin_associated_unit
    unit_key[k] = pulp_unit['metadata'].pop(k)
PulpExecutionException: Pulp exception occurred: PulpExecutionException
2013-06-13 14:34:02,600 pulp.server.dispatch.task:INFO: FAILURE: Task 54cf6b3f-3d2b-43ea-ac0a-8e57c27da880: CallRequest: RepoUnitAssociationManager.associate_from_repo('pulp', u'pulp-copy', import_config_override=****, criteria=****)

Comment 1 Barnaby Court 2013-09-11 13:21:16 UTC
Preethi, I tested this using both pulp-2.2 and master branches with the following script and was unable to duplicate the issue.  Can you verify?

pulp-admin rpm repo create --repo-id pulp --feed http://repos.fedorapeople.org/repos/pulp/pulp/testing/fedora-19/x86_64/
pulp-admin rpm repo sync run --repo-id pulp
pulp-admin rpm repo create --repo-id pulp-copy 
pulp-admin rpm repo copy rpm --from-repo-id pulp --to-repo-id pulp-copy --str-eq "name=pulp-server" --recursive
pulp-admin rpm repo copy group --from-repo-id pulp --to-repo-id pulp-copy --str-eq "id=pulp-server" --recursive 



:: Results of Script ::

[bcourt@bcourt ci]$ pulp-admin rpm repo create --repo-id pulp --feed http://repos.fedorapeople.org/repos/pulp/pulp/testing/fedora-19/x86_64/
Successfully created repository [pulp]

[bcourt@bcourt ci]$ pulp-admin rpm repo sync run --repo-id pulp
+----------------------------------------------------------------------+
                    Synchronizing Repository [pulp]
+----------------------------------------------------------------------+

This command may be exited by pressing ctrl+c without affecting the actual
operation on the server.

Downloading metadata...
[\]
... completed

Downloading repository content...
[==================================================] 100%
RPMs:       42/42 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
... completed

Importing errata...
[-]
... completed

Importing package groups/categories...
[-]
... completed

Publishing packages...
[==================================================] 100%
Packages: 42/42 items
... completed

Publishing distributions...
[==================================================] 100%
Distributions: 0/0 items
... completed

Generating metadata
[\]
... completed

Publishing repository over HTTPS
[-]
... completed

[bcourt@bcourt ci]$ pulp-admin rpm repo create --repo-id pulp-copy 
Successfully created repository [pulp-copy]

[bcourt@bcourt ci]$ pulp-admin rpm repo copy rpm --from-repo-id pulp --to-repo-id pulp-copy --str-eq "name=pulp-server" --recursive
This command may be exited via ctrl+c without affecting the request.

[-]
Waiting to begin...

[-]
Running...

Units:
  pulp-server-2.3.0-0.8.alpha.fc19-noarch

[bcourt@bcourt ci]$ pulp-admin rpm repo copy group --from-repo-id pulp --to-repo-id pulp-copy --str-eq "id=pulp-server" --recursive
This command may be exited via ctrl+c without affecting the request.

[-]
Waiting to begin...

[-]
Running...

Units:
  pulp-server

[bcourt@bcourt ci]$

Comment 2 Jeff Ortel 2013-09-12 22:35:58 UTC
build: 2.3.0-0.10.alpha

Comment 3 Preethi Thomas 2013-09-19 20:14:01 UTC
verified that this has been fixed
[root@pulp-v2-server ~]# rpm -qa pulp-server
pulp-server-2.3.0-0.14.alpha.el6.noarch
[root@pulp-v2-server ~]# 

[root@pulp-v2-server ~]# pulp-admin rpm repo create --repo-id pulp --feed http://repos.fedorapeople.org/repos/pulp/pulp/v2/testing/fedora-19/x86_64/
Successfully created repository [pulp]

[root@pulp-v2-server ~]# pulp-admin rpm repo sync --repo-id pulp
Usage: pulp-admin [SUB_SECTION, ..] COMMAND
Description: run, schedule, or view the status of sync tasks

Available Sections:
  schedules - manage repository sync schedules

Available Commands:
  run    - triggers an immediate sync of a repository
  status - displays the status of a repository's sync tasks
[root@pulp-v2-server ~]# pulp-admin rpm repo sync run --repo-id pulp
+----------------------------------------------------------------------+
                    Synchronizing Repository [pulp]
+----------------------------------------------------------------------+

This command may be exited by pressing ctrl+c without affecting the actual
operation on the server.

Downloading metadata...
[\]
... completed

Downloading repository content...
[==================================================] 100%
RPMs:       42/42 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
... completed

Importing errata...
[-]
... completed

Importing package groups/categories...
[-]
... completed

Publishing packages...
[==================================================] 100%
Packages: 42/42 items
... completed

Publishing distributions...
[==================================================] 100%
Distributions: 0/0 items
... completed

Generating metadata
[\]
... completed

Publishing repository over HTTPS
[-]
... completed

[root@pulp-v2-server ~]# pulp-admin rpm repo create --repo-id pulp-copy
Successfully created repository [pulp-copy]

[root@pulp-v2-server ~]#  pulp-admin rpm repo copy rpm  --from-repo-id pulp --to-repo-id pulp-copy --str-eq "name=pulp-server" --recursive
This command may be exited via ctrl+c without affecting the request.

[-]
Waiting to begin...

[-]
Running...

Units:
  m2crypto-0.21.1.pulp-8.fc19-x86_64
  mod_wsgi-3.4-1.pulp.fc19-x86_64
  pulp-server-2.3.0-0.14.alpha.fc19-noarch
  python-gofer-0.76-1.fc19-noarch
  python-isodate-0.5.0-1.pulp.fc19-noarch
  python-nectar-1.1.1-1.fc19-noarch
  python-oauth2-1.5.170-3.pulp.fc19-noarch
  python-okaara-1.0.32-1.fc19-noarch
  python-qpid-0.7.946106-2.fc19-noarch
  python-webpy-0.32-9.fc19-noarch

[root@pulp-v2-server ~]# pulp-admin rpm repo copy group --from-repo-id pulp --to-repo-id pulp-copy --str-eq "id=pulp-server" --recursive 
This command may be exited via ctrl+c without affecting the request.

[-]
Waiting to begin...

[-]
Running...

Units:
 package_group:
  pulp-server
 rpm:
  createrepo-0.9.9-21.2.pulp.fc19-noarch
  pulp-puppet-plugins-2.3.0-0.14.alpha.fc19-noarch
  pulp-rpm-plugins-2.3.0-0.14.alpha.fc19-noarch
  pulp-selinux-2.3.0-0.14.alpha.fc19-noarch
  python-rhsm-1.8.0-1.pulp.fc19-x86_64
  python-semantic-version-2.2.0-3.fc19-noarch

[root@pulp-v2-server ~]#

Comment 4 Preethi Thomas 2013-12-09 14:31:36 UTC
Pulp 2.3 released.