Bug 1515195 - generate errata applicability task occasionally raises duplicatekey error [6.2.z]
Summary: generate errata applicability task occasionally raises duplicatekey error [6....
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.3.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: Roman Plevka
URL:
Whiteboard:
Depends On: 1468022
Blocks: 1122832
TreeView+ depends on / blocked
 
Reported: 2017-11-20 11:29 UTC by Sachin Ghai
Modified: 2023-03-24 13:54 UTC (History)
42 users (show)

Fixed In Version: pulp-2.8.7.18-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1468022
: 1530687 (view as bug list)
Environment:
Last Closed: 2018-02-05 13:55:32 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Pulp Redmine 2874 0 High CLOSED - CURRENTRELEASE Race condition during applicability regeneration for consumers with same profile 2017-11-20 13:47:23 UTC
Red Hat Knowledge Base (Solution) 3176371 0 None None None 2017-11-20 11:29:50 UTC
Red Hat Product Errata RHSA-2018:0273 0 normal SHIPPED_LIVE Important: Red Hat Satellite 6 security, bug fix, and enhancement update 2018-02-08 00:35:29 UTC

Comment 1 Sachin Ghai 2017-11-20 11:32:10 UTC
Hit same issue on upgrading 6.2.12 -> 6.3 snap25

Nov 17 16:00:01 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (4270-61088) Traceback (most recent call last):
Nov 17 16:00:01 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (4270-61088)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Nov 17 16:00:01 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (4270-61088)     R = retval = fun(*args, **kwargs)
Nov 17 16:00:01 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (4270-61088)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 473, in __call__
Nov 17 16:00:01 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (4270-61088)     return super(Task, self).__call__(*args, **kwargs)
Nov 17 16:00:01 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (4270-61088)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in __call__
Nov 17 16:00:01 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (4270-61088)     return super(PulpTask, self).__call__(*args, **kwargs)
Nov 17 16:00:01 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (4270-61088)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Nov 17 16:00:01 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (4270-61088)     return self.run(*args, **kwargs)
Nov 17 16:00:01 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (4270-61088)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/consumer/applicability.py", line 107, in regenerate_applicability_for_consumers
Nov 17 16:00:01 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (4270-61088)     manager.regenerate_applicability(profile_hash, content_type, profile_id, repo_id)
Nov 17 16:00:01 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (4270-61088)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/consumer/applicability.py", line 272, in regenerate_applicability
Nov 17 16:00:01 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (4270-61088)     applicability)
Nov 17 16:00:01 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (4270-61088)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/consumer/applicability.py", line 384, in create
Nov 17 16:00:01 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (4270-61088)     applicability.save()
Nov 17 16:00:01 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (4270-61088)   File "/usr/lib/python2.7/site-packages/pulp/server/db/model/consumer.py", line 234, in save
Nov 17 16:00:01 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (4270-61088)     self._id = self.get_collection().insert(new_document)
Nov 17 16:00:01 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (4270-61088)   File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 2200, in insert
Nov 17 16:00:01 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (4270-61088)     check_keys, manipulate, write_concern)
Nov 17 16:00:01 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (4270-61088)   File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 530, in _insert
Nov 17 16:00:01 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (4270-61088)     check_keys, manipulate, write_concern, op_id, bypass_doc_val)
Nov 17 16:00:01 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (4270-61088)   File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 513, in _insert_one
Nov 17 16:00:01 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (4270-61088)     _check_write_command_response([(0, result)])
Nov 17 16:00:01 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (4270-61088)   File "/usr/lib64/python2.7/site-packages/pymongo/helpers.py", line 300, in _check_write_command_response
Nov 17 16:00:01 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (4270-61088)     raise DuplicateKeyError(error.get("errmsg"), 11000, error)
Nov 17 16:00:01 qe-sat6-upgrade-rhel7 pulp: celery.worker.job:ERROR: (4270-61088) DuplicateKeyError: insertDocument :: caused by :: 11000 E11000 duplicate key error index: pulp_database.repo_profile_applicability.$profile_hash_-1_repo_id_-1  dup key: { : "0b583821e55e8f92bb3c7b932740bb02b0819726d7c2dcfb835a4c44b226ed21", : "Default_Organization-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6Server" }

Comment 2 pulp-infra@redhat.com 2017-11-20 13:47:24 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 3 pulp-infra@redhat.com 2017-11-20 13:47:33 UTC
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.

Comment 4 pulp-infra@redhat.com 2017-11-20 14:10:31 UTC
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.

Comment 8 Roman Plevka 2018-01-16 16:32:46 UTC
VERIFIED
on sat6.2.14-1


- the duplicate key error is now handled properly


- prepare 2 consumers (register them to a sat, have some repos with packages and errata)
- possibly tweak the pulp server.managers.consumer.applicability code to add some time.sleep() just after the regenerate_applicability function call that would make your opportunity window wider.
- also add some _logger.info() messages into the `expect DuplicateKeyError` branch in order to make sure, you're replicated the behavior correctly
(you might need to tweak pulp.server.logs to whitelist the logger)

- now send out a package profile for a host1 (that would contain package version, triggering the errata status)
- while you're hooked on the sleep(), trigger package profile upload from the other host

- after the sleep() from the first package profile request expires, pulp should create an Applicability Object.
- after the sleep() from the second package profile request expires, it should follow the same route of creating the applicability object, however it should raise the DuplicateKeyError, due this one already exists. However, the error is properly caught and handled.

Comment 12 errata-xmlrpc 2018-02-05 13:55:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:0273


Note You need to log in before you can comment on or make changes to this bug.