Bug 843800 - (katello-dotpackage) Syncronization raises an exception when package have a different name structure
Syncronization raises an exception when package have a different name structure
Status: CLOSED CURRENTRELEASE
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Installer (Show other bugs)
Nightly
All Linux
high Severity high (vote)
: Unspecified
: 6.0
Assigned To: Lukas Zapletal
Katello QA List
: Regression, Triaged
: 879679 (view as bug list)
Depends On:
Blocks: 853356
  Show dependency treegraph
 
Reported: 2012-07-27 07:54 EDT by Marcelo Moreira de Mello
Modified: 2014-09-18 11:36 EDT (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 853356 (view as bug list)
Environment:
Last Closed: 2014-01-21 14:54:41 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Marcelo Moreira de Mello 2012-07-27 07:54:42 EDT
Description of problem:

Syncronization raises an exception when package have a different name structure. 


After entering the repository for F16 (http://fedora.c3sl.ufpr.br/linux/releases/16/Fedora/x86_64/os/), when finishing the sync process, it raises the exception below:

==> pulp/grinder.log <==
2012-07-27 03:08:16,728 1020:140280250484480: grinder.ParallelFetch:INFO: ParallelFetch:138 1 threads are active. 1 items left to be fetched
2012-07-27 03:08:16,728 1020:140280250484480: grinder.ParallelFetch:INFO: ParallelFetch:329 WorkerThread deleting ActiveObject
2012-07-27 03:08:16,732 1020:140280250484480: grinder.ParallelFetch:INFO: ParallelFetch:344 Thread ending
2012-07-27 03:08:17,219 1020:140280258877184: grinder.ParallelFetch:INFO: ParallelFetch:234 All threads have finished.
2012-07-27 03:08:17,270 1020:140280258877184: grinder.ParallelFetch:INFO: ParallelFetch:253 ParallelFetch: 3113 items successfully processed, 3113 downloaded, 0 items had errors
2012-07-27 03:08:17,271 1020:140280258877184: grinder.ParallelFetch:INFO: ParallelFetch:258 Transferred [564667992] bytes of [tree_file]
2012-07-27 03:08:17,271 1020:140280258877184: grinder.ParallelFetch:INFO: ParallelFetch:258 Transferred [3455015673] bytes of [rpm]
2012-07-27 03:08:17,272 1020:140280258877184: grinder.ParallelFetch:INFO: ParallelFetch:259 Transferred [4019683665] total bytes in 2242.64618111 seconds
2012-07-27 03:08:17,272 1020:140280258877184: grinder.RepoFetch:INFO: RepoFetch:246 Finalizing metadata, moving /var/lib/pulp//repos/mmello_ORG/Library/custom/Fedora_16_x86_64/fedora_ufpr//repodata.new to /var/lib/pulp//repos/mmello_ORG/Library/custom/Fedora_16_x86_64/fedora_ufpr//repodata
2012-07-27 03:08:17,392 1020:140280258877184: grinder.RepoFetch:INFO: RepoFetch:182 Cleaning any orphaned packages..
2012-07-27 03:08:18,491 1020:140280258877184: grinder.RepoFetch:INFO: RepoFetch:192 Processed <>,<http://fedora.c3sl.ufpr.br/linux/releases/16/Fedora/x86_64/os/> with <0> items in [2223] seconds. Report: 3113 successes, 3113 downloads, 0 errors
2012-07-27 03:08:18,492 1020:140280258877184: grinder.ParallelFetch:INFO: ParallelFetch:203 Grinder stopping
2012-07-27 03:08:18,492 1020:140280258877184: grinder.ParallelFetch:INFO: ParallelFetch:287 stop() invoked
2012-07-27 03:08:18,492 1020:140280258877184: grinder.ParallelFetch:INFO: ParallelFetch:297 stop() completed
2012-07-27 03:08:18,493 1020:140280258877184: grinder.ParallelFetch:INFO: ParallelFetch:207 Told thread <<WorkerThread(Thread-20, stopped daemon 140280267269888)>> to stop
2012-07-27 03:08:18,493 1020:140280258877184: grinder.ParallelFetch:INFO: ParallelFetch:287 stop() invoked
2012-07-27 03:08:18,494 1020:140280258877184: grinder.ParallelFetch:INFO: ParallelFetch:297 stop() completed
2012-07-27 03:08:18,494 1020:140280258877184: grinder.ParallelFetch:INFO: ParallelFetch:207 Told thread <<WorkerThread(Thread-21, stopped daemon 140280275662592)>> to stop
2012-07-27 03:08:18,494 1020:140280258877184: grinder.ParallelFetch:INFO: ParallelFetch:287 stop() invoked
2012-07-27 03:08:18,495 1020:140280258877184: grinder.ParallelFetch:INFO: ParallelFetch:297 stop() completed
2012-07-27 03:08:18,495 1020:140280258877184: grinder.ParallelFetch:INFO: ParallelFetch:207 Told thread <<WorkerThread(Thread-22, stopped daemon 140280250484480)>> to stop
2012-07-27 03:08:18,495 1020:140280258877184: grinder.ParallelFetch:INFO: ParallelFetch:287 stop() invoked
2012-07-27 03:08:18,496 1020:140280258877184: grinder.ParallelFetch:INFO: ParallelFetch:297 stop() completed
2012-07-27 03:08:18,496 1020:140280258877184: grinder.ParallelFetch:INFO: ParallelFetch:207 Told thread <<WorkerThread(Thread-23, stopped daemon 140280242091776)>> to stop

==> pulp/pulp.log <==
2012-07-27 03:08:18,497 1020:140280258877184: pulp.server.api.synchronizers:INFO: synchronizers:793 YumSynchronizer reported 3113 successes, 3113 downloads, 0 errors
2012-07-27 03:10:08,403 1020:140280258877184: pulp.server.api.synchronizers:INFO: synchronizers:856 checksum type for repo id mmello_ORG-Fedora_16_x86_64-fedora_ufpr is sha256
2012-07-27 03:10:08,413 1020:140280258877184: pulp.server.api.synchronizers:INFO: synchronizers:839 Running createrepo, this may take a few minutes to complete.
2012-07-27 03:10:08,417 1020:140280258877184: pulp.server.util:INFO: util:507 metadata found; taking backup.
2012-07-27 03:10:09,050 1020:140280258877184: pulp.server.util:INFO: util:489 started repo metadata update: ['createrepo', '--database', '--checksum', 'sha256', '-g', '/var/lib/pulp//repos/mmello_ORG/Library/custom/Fedora_16_x86_64/fedora_ufpr/repodata/d7017d949b6f30725d20ee3ef45d9fa03d603464b9d411e056af2fddf5ceb608-Fedora-16-comps.xml', '--update', '/var/lib/pulp//repos/mmello_ORG/Library/custom/Fedora_16_x86_64/fedora_ufpr']
2012-07-27 03:15:08,206 1020:140280258877184: pulp.server.util:INFO: util:542 createrepo on /var/lib/pulp//repos/mmello_ORG/Library/custom/Fedora_16_x86_64/fedora_ufpr finished
2012-07-27 03:15:08,226 1020:140280258877184: pulp.server.api.synchronizers:INFO: synchronizers:843 Createrepo finished in 299.809221983 seconds
2012-07-27 03:15:08,307 1020:140280258877184: pulp.server.api.synchronizers:INFO: synchronizers:477 Created a distributionID ks-Fedora-Fedora-16-x86_64
2012-07-27 03:15:08,331 1020:140280258877184: pulp.server.api.synchronizers:INFO: synchronizers:422 No metadata for 'File Sync' present; no files to import to repo..
2012-07-27 03:15:08,340 1020:140280258877184: pulp.server.api.synchronizers:INFO: synchronizers:385 Loading comps group info from: /var/lib/pulp//repos/mmello_ORG/Library/custom/Fedora_16_x86_64/fedora_ufpr/repodata/d7017d949b6f30725d20ee3ef45d9fa03d603464b9d411e056af2fddf5ceb608-d7017d949b6f30725d20ee3ef45d9fa03d603464b9d411e056af2fddf5ceb608-Fedora-16-comps.xml
2012-07-27 03:15:08,704 1020:140280258877184: pulp.server.api.synchronizers:INFO: synchronizers:412 Skipping errata imports from sync process
2012-07-27 03:15:08,981 1020:140280258877184: pulp.server.tasking.task:ERROR: task:472 Task failed: Task 3c3fe154-d7ac-11e1-83db-525400623320: _sync(mmello_ORG-Fedora_16_x86_64-fedora_ufpr, synchronizer=<pulp.server.api.synchronizers.YumSynchronizer object at 0x7f95a4041650>, skip={}, max_speed=None, threads=4, progress_callback=<bound method RepoSyncTask.progress_callback of <pulp.server.api.repo_sync_task.RepoSyncTask object at 0x7f9584023110>>)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pulp/server/tasking/task.py", line 418, in run
    result = self.callable(*self.args, **self.kwargs)
  File "/usr/lib/python2.7/site-packages/pulp/server/api/repo_sync.py", line 283, in _sync
    progress_callback, synchronizer, max_speed, threads)
  File "/usr/lib/python2.7/site-packages/pulp/server/api/repo_sync.py", line 379, in fetch_content
    added_errataids = synchronizer.import_metadata(repo_dir, repo_id, skip_dict)
  File "/usr/lib/python2.7/site-packages/pulp/server/api/synchronizers.py", line 415, in import_metadata
    self.repo_api.collection.save(repo, safe=True)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/connection.py", line 80, in retry
    return method(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 237, in save
    manipulate, safe, _check_keys=True, **kwargs)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/connection.py", line 80, in retry
    return method(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 411, in update
    _check_keys, self.__uuid_subtype), safe)
InvalidDocument: key 'openoffice.org-voikko' must not contain '.'



Version-Release number of selected component (if applicable):
katello-0.2.50-1.fc16.noarch
pulp-1.1.11-1.fc16.noarch



How reproducible:
100%


Steps to Reproduce:
1. install a f16 box
2. set the repository for http://fedora.c3sl.ufpr.br/linux/releases/16/Fedora/x86_64/os/
3. sync the channel
  
Actual results:

  sync raises an error


Expected results:

  sync repo as expected


Additional info:
Comment 1 Marcelo Moreira de Mello 2012-07-27 07:58:52 EDT

[root@katello ~]# find / -iname "*openoffice.org-vo*"
/var/lib/pulp/packages/openoffice.org-voikko
/var/lib/pulp/packages/openoffice.org-voikko/3.1.2/5.fc16/x86_64/eb154dbedf52c544e783e0af679cc16d1ef290107900efd606951474c6c1533b/openoffice.org-voikko-3.1.2-5.fc16.x86_64.rpm
/var/lib/pulp/repos/mmello_ORG/Library/custom/Fedora_16_x86_64/fedora_ufpr/Packages/openoffice.org-voikko-3.1.2-5.fc16.x86_64.rpm
[root@katello ~]# rpm -q katello pulp
katello-0.2.50-1.fc16.noarch
pulp-1.1.11-1.fc16.noarch
[root@katello ~]#
Comment 2 Marcelo Moreira de Mello 2012-07-27 08:00:06 EDT
[root@katello ~]# rpm -q pymongo
pymongo-2.1.1-1.fc16.x86_64
Comment 3 Marcelo Moreira de Mello 2012-07-31 16:53:42 EDT

  To be able to sync the repository for F16, you must downgrade the pymongo version to pymongo-1.11-1.fc16.x86_64. 

   # yum downgrade pymongo python-bson

   # rpm -q pymongo python-bson
   pymongo-1.11-1.fc16.x86_64
   python-bson-1.11-1.fc16.x86_64


   Afterwards, restart katello services and sync again. 


=> pulp/pulp.log <==
2012-07-31 17:45:48,759 1093:140043599451904: pulp.server.api.synchronizers:INFO: synchronizers:856 checksum type for repo id mmelloORG-Fedora_16_x86_64-fedora_ufpr is sha256
2012-07-31 17:45:48,798 1093:140043599451904: pulp.server.api.synchronizers:INFO: synchronizers:839 Running createrepo, this may take a few minutes to complete.
2012-07-31 17:45:48,798 1093:140043599451904: pulp.server.util:INFO: util:507 metadata found; taking backup.
2012-07-31 17:45:49,070 1093:140043599451904: pulp.server.util:INFO: util:489 started repo metadata update: ['createrepo', '--database', '--checksum', 'sha256', '-g', '/var/lib/pulp//repos/mmelloORG/Library/custom/Fedora_16_x86_64/fedora_ufpr/repodata/d7017d949b6f30725d20ee3ef45d9fa03d603464b9d411e056af2fddf5ceb608-Fedora-16-comps.xml', '--update', '/var/lib/pulp//repos/mmelloORG/Library/custom/Fedora_16_x86_64/fedora_ufpr']
2012-07-31 17:49:28,474 1093:140043599451904: pulp.server.util:INFO: util:542 createrepo on /var/lib/pulp//repos/mmelloORG/Library/custom/Fedora_16_x86_64/fedora_ufpr finished
2012-07-31 17:49:28,478 1093:140043599451904: pulp.server.api.synchronizers:INFO: synchronizers:843 Createrepo finished in 219.679879904 seconds
2012-07-31 17:49:28,514 1093:140043599451904: pulp.server.api.distribution:INFO: distribution:50 Distribution with id ks-Fedora-Fedora-16-x86_64 already exists
2012-07-31 17:49:28,548 1093:140043599451904: pulp.server.api.synchronizers:INFO: synchronizers:422 No metadata for 'File Sync' present; no files to import to repo..
2012-07-31 17:49:28,557 1093:140043599451904: pulp.server.api.synchronizers:INFO: synchronizers:385 Loading comps group info from: /var/lib/pulp//repos/mmelloORG/Library/custom/Fedora_16_x86_64/fedora_ufpr/repodata/d7017d949b6f30725d20ee3ef45d9fa03d603464b9d411e056af2fddf5ceb608-d7017d949b6f30725d20ee3ef45d9fa03d603464b9d411e056af2fddf5ceb608-Fedora-16-comps.xml
2012-07-31 17:49:28,873 1093:140043599451904: pulp.server.api.synchronizers:INFO: synchronizers:412 Skipping errata imports from sync process
2012-07-31 17:49:29,185 1093:140043599451904: pulp.server.api.repo_sync:INFO: repo_sync:286 Sync on mmelloORG-Fedora_16_x86_64-fedora_ufpr returned 3107 packages, 0 errata in 537.659252882 seconds
2012-07-31 17:49:29,525 1093:140043599451904: pulp.server.api.repo_sync:INFO: repo_sync:297 0 old packages to process, 3107 new packages to process
2012-07-31 17:49:29,531 1093:140043599451904: pulp.server.api.repo_sync:INFO: repo_sync:299 Removing 0 old packages
2012-07-31 17:49:29,964 1093:140043599451904: pulp.server.api.repo_sync:INFO: repo_sync:305 Adding 3107 new packages
2012-07-31 17:49:30,806 1093:140043599451904: pulp.server.api.repo_sync:INFO: repo_sync:314 Examining 0 errata from repo mmelloORG-Fedora_16_x86_64-fedora_ufpr
2012-07-31 17:49:31,292 1093:140043599451904: pulp.server.api.repo_sync:INFO: repo_sync:318 Removing 0 old errata from repo mmelloORG-Fedora_16_x86_64-fedora_ufpr
2012-07-31 17:49:32,512 1093:140043599451904: pulp.server.api.repo_sync:INFO: repo_sync:327 Adding 0 new errata to repo mmelloORG-Fedora_16_x86_64-fedora_ufpr
2012-07-31 17:49:32,958 1093:140043599451904: pulp.server.tasking.task:INFO: task:454 Task succeeded: Task f870a2c2-db4f-11e1-bbfd-525400d00a36: _sync(mmelloORG-Fedora_16_x86_64-fedora_ufpr, synchronizer=<pulp.server.api.synchronizers.YumSynchronizer object at 0x7f5eb162c290>, skip={}, max_speed=None, threads=4, progress_callback=<bound method RepoSyncTask.progress_callback of <pulp.server.api.repo_sync_task.RepoSyncTask object at 0x7f5eb162c910>>)

 webUI will display the sync complete message.
Comment 4 Ivan Necas 2012-08-31 05:05:48 EDT
Note: the same issue is when using EPEL
Comment 6 Mike McCune 2012-09-11 19:40:54 EDT
On EL6 I tried with the pymongo from EPEL and it synced fine:

$ rpm -q redhat-release-server-6Server pymongo pulp
redhat-release-server-6Server-6.3.0.3.el6.x86_64
pymongo-2.1.1-1.el6.x86_64
pulp-1.1.11-1.el6.noarch

$ pulp-admin content list --repoid=ACME_Corporation-Random-random-2 |grep open
openoffice.org-voikko-3.1.2-5.fc16.x86_64.rpm,eb154dbedf52c544e783e0af679cc16d1ef290107900efd606951474c6c1533b

no errors in grinder.log or webUI
Comment 7 Marcelo Moreira de Mello 2012-09-12 16:51:06 EDT
(In reply to comment #6)
> On EL6 I tried with the pymongo from EPEL and it synced fine:
> 
> $ rpm -q redhat-release-server-6Server pymongo pulp
> redhat-release-server-6Server-6.3.0.3.el6.x86_64
> pymongo-2.1.1-1.el6.x86_64
> pulp-1.1.11-1.el6.noarch
> 
> $ pulp-admin content list --repoid=ACME_Corporation-Random-random-2 |grep
> open
> openoffice.org-voikko-3.1.2-5.fc16.x86_64.rpm,
> eb154dbedf52c544e783e0af679cc16d1ef290107900efd606951474c6c1533b
> 
> no errors in grinder.log or webUI

 Hello Mike, 

   Are you able to sync it using the if you Katello box is a F16?

   I hit this issue this morning again and had to downgrade the pymongo to pymongo-1.11-1.fc16.x86_64 to be able to sync the package. 

   Thanks. 
mmello
Comment 8 Mike McCune 2012-11-28 11:15:19 EST
*** Bug 879679 has been marked as a duplicate of this bug. ***
Comment 9 David Kaylor 2012-12-03 10:09:44 EST
The same issue occurs when syncing the "Red Hat Enterprise Linux 6 Server - Optional RPMs x86_64 6.3" repository:

key 'openoffice.org-langpack-en' must not contain '.'
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/server/tasking/task.py", line 418, in run
    result = self.callable(*self.args, **self.kwargs)
  File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 283, in _sync
    progress_callback, synchronizer, max_speed, threads)
  File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 379, in fetch_content
    added_errataids = synchronizer.import_metadata(repo_dir, repo_id, skip_dict)
  File "/usr/lib/python2.6/site-packages/pulp/server/api/synchronizers.py", line 416, in import_metadata
    self.repo_api.collection.save(repo, safe=True)
  File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py", line 80, in retry
    return method(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", line 237, in save
    manipulate, safe, _check_keys=True, **kwargs)
  File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py", line 80, in retry
    return method(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", line 411, in update
    _check_keys, self.__uuid_subtype), safe)
InvalidDocument: key 'openoffice.org-langpack-en' must not contain '.'

The Katello server is also RHEL 6.3 and the pymongo package installed is pymongo-2.1.1-1.el6.x86_64.
Comment 10 Bryan Kearney 2014-01-21 14:07:38 EST
Moving to Sat6 to be tracked there. Upstream bugs are moving to redmine.

Note You need to log in before you can comment on or make changes to this bug.