Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1488062

Summary: Yum update on multiple client gives DuplicateKeyError when doing pulp_database.repo_profile_applicability
Product: Red Hat Satellite Reporter: Ben <ben.argyle>
Component: PulpAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED DUPLICATE QA Contact: Katello QA List <katello-qa-list>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.2.10CC: ttereshc
Target Milestone: Unspecified   
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-09-11 20:43:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ben 2017-09-04 08:45:19 UTC 
Description of problem:
I'm wondering if this related to BZ 1468022.  Whenever I join and yum update a number of content hosts (via Ansible, so they're pretty much simultaneous) to Satellite I get duplicate key errors:

Sep  4 09:26:42 satellite1 pulp: celery.worker.job:ERROR: (19163-81920) DuplicateKeyError: insertDocument :: caused by :: 11000 E11000 duplicate key error index: pulp_database.repo_profile_applicability.$profile_hash_-1_repo_id_-1  dup key: { : "3bf26d7af56eaddb58d4842aae6cc87201f46dce5ebbd2e45177292f32ab64b6", : "UIS-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_7_Server_RPMs_x86_64_7Server" }
Sep  4 09:26:42 satellite1 pulp: celery.worker.job:ERROR: (19160-36736) Task pulp.server.managers.consumer.applicability.regenerate_applicability_for_consumers[cb22926b-7e1f-4738-b284-04c03fe002f3] raised unexpected: DuplicateKeyError(u'insertDocument :: caused by :: 11000 E11000 duplicate key error index: pulp_database.repo_profile_applicability.$profile_hash_-1_repo_id_-1  dup key: { : "3bf26d7af56eaddb58d4842aae6cc87201f46dce5ebbd2e45177292f32ab64b6", : "UIS-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_7_Server_RPMs_x86_64_7Server" }',)
Sep  4 09:26:42 satellite1 pulp: celery.worker.job:ERROR: (19160-36736) Traceback (most recent call last):
Sep  4 09:26:42 satellite1 pulp: celery.worker.job:ERROR: (19160-36736)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Sep  4 09:26:42 satellite1 pulp: celery.worker.job:ERROR: (19160-36736)     R = retval = fun(*args, **kwargs)
Sep  4 09:26:42 satellite1 pulp: celery.worker.job:ERROR: (19160-36736)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 473, in __call__
Sep  4 09:26:42 satellite1 pulp: celery.worker.job:ERROR: (19160-36736)     return super(Task, self).__call__(*args, **kwargs)
Sep  4 09:26:42 satellite1 pulp: celery.worker.job:ERROR: (19160-36736)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in __call__
Sep  4 09:26:42 satellite1 pulp: celery.worker.job:ERROR: (19160-36736)     return super(PulpTask, self).__call__(*args, **kwargs)
Sep  4 09:26:42 satellite1 pulp: celery.worker.job:ERROR: (19160-36736)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Sep  4 09:26:42 satellite1 pulp: celery.worker.job:ERROR: (19160-36736)     return self.run(*args, **kwargs)
Sep  4 09:26:42 satellite1 pulp: celery.worker.job:ERROR: (19160-36736)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/consumer/applicability.py", line 107, in regenerate_applicability_for_consumers
Sep  4 09:26:42 satellite1 pulp: celery.worker.job:ERROR: (19160-36736)     manager.regenerate_applicability(profile_hash, content_type, profile_id, repo_id)
Sep  4 09:26:42 satellite1 pulp: celery.worker.job:ERROR: (19160-36736)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/consumer/applicability.py", line 272, in regenerate_applicability
Sep  4 09:26:42 satellite1 pulp: celery.worker.job:ERROR: (19160-36736)     applicability)
Sep  4 09:26:42 satellite1 pulp: celery.worker.job:ERROR: (19160-36736)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/consumer/applicability.py", line 384, in create
Sep  4 09:26:42 satellite1 pulp: celery.worker.job:ERROR: (19160-36736)     applicability.save()
Sep  4 09:26:42 satellite1 pulp: celery.worker.job:ERROR: (19160-36736)   File "/usr/lib/python2.7/site-packages/pulp/server/db/model/consumer.py", line 234, in save
Sep  4 09:26:42 satellite1 pulp: celery.worker.job:ERROR: (19160-36736)     self._id = self.get_collection().insert(new_document)
Sep  4 09:26:42 satellite1 pulp: celery.worker.job:ERROR: (19160-36736)   File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 2200, in insert
Sep  4 09:26:42 satellite1 pulp: celery.worker.job:ERROR: (19160-36736)     check_keys, manipulate, write_concern)
Sep  4 09:26:42 satellite1 pulp: celery.worker.job:ERROR: (19160-36736)   File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 530, in _insert
Sep  4 09:26:42 satellite1 pulp: celery.worker.job:ERROR: (19160-36736)     check_keys, manipulate, write_concern, op_id, bypass_doc_val)
Sep  4 09:26:42 satellite1 pulp: celery.worker.job:ERROR: (19160-36736)   File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 513, in _insert_one
Sep  4 09:26:42 satellite1 pulp: celery.worker.job:ERROR: (19160-36736)     _check_write_command_response([(0, result)])
Sep  4 09:26:42 satellite1 pulp: celery.worker.job:ERROR: (19160-36736)   File "/usr/lib64/python2.7/site-packages/pymongo/helpers.py", line 300, in _check_write_command_response
Sep  4 09:26:42 satellite1 pulp: celery.worker.job:ERROR: (19160-36736)     raise DuplicateKeyError(error.get("errmsg"), 11000, error)

I'm unsure how to stop this from happening... unless this specifically is fixed in 6.2.11.


Version-Release number of selected component (if applicable):
Satellite 6.2.10

How reproducible:
Whenever two or more content hosts are registered with Satellite and then yum updated.

Steps to Reproduce:
1. Join hosts to Satellite
2. Issue yum update command
3. See errors

Actual results:

Everything _seems_ to work on the client end, but I'm concerned about databases errors.


Expected results:

Everything should just work, without duplicate keys.


Additional info:

I'm willing to upgrade to 6.2.11, but would appreciate being told whether my database is inconsistent because of all of the duplicate keys.  Thank you.
Comment 1 Ben 2017-09-04 16:21:03 UTC 
OK, I kept getting the same errors every time pulp_database.repo_profile_applicability fired for four of the six content hosts I subscribed at the same time with an Ansible role.  I cleaned them out of Satellite and then resubbed them, manually, with a few seconds between each one.  No more duplicate keys.  Disappointing that I had to do this.  Hopefully this is one of the bugs that's fixed in 6.2.11.
Comment 2 Tanya Tereshchenko 2017-09-11 20:43:15 UTC 
Hi Ben,
You found the right BZ 1468022, it's the same issue: race condition for applicability tasks. It can happen in a few cases, including yours, when multiple content hosts are registered at the same time with the same set of packages installed on them. It is not fixed in 6.2.11, as far as I know.

In case of any other questions relevant to this issue, please, comment on BZ 1468022.

Thanks!

*** This bug has been marked as a duplicate of bug 1468022 ***