Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1367758 - [Sat6.2 upgrade] pulp-manage-db creates duplicate records in units_ostree
Summary: [Sat6.2 upgrade] pulp-manage-db creates duplicate records in units_ostree
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.2.0
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: Katello QA List
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-08-17 12:00 UTC by Pavel Moravec
Modified: 2019-12-16 06:23 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-08-18 20:32:30 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1374723 0 high CLOSED [Improvement] prevent running pulp-manage-db when pulp workers are up 2021-04-06 17:56:01 UTC
Red Hat Knowledge Base (Solution) 2534911 0 None None None 2016-08-18 20:32:15 UTC

Internal Links: 1374723

Description Pavel Moravec 2016-08-17 12:00:37 UTC 
Description of problem:
Upgrade of Satellite to 6.2 fails on step pulp-manage-db with backtrace suggesting units_ostree collection violates unique key/index.

Manually removing duplicates (and references to them) and re-running just pulp-manage-db fails the same way.


Version-Release number of selected component (if applicable):
Sat 6.1.9 -> 6.2 upgrade
pulp-server 2.6.0.21-1 -> 2.8.3.3-1 upgrade


How reproducible:
??? I guess 100%


Steps to Reproduce:
n.a. at the moment, I guess it is enough to sync an ostree unit in 6.1, put to a repo, publish the repo, and upgrade to 6.2


Actual results:
upgrade fails with:

E11000 duplicate key error index: pulp_database.units_ostree.$remote_id_1_branch_1_commit_1  dup key: { : "174b6356ebe246768ac1600709cfefc6589a43503397a9e7589111d4da9188f6", : "rhel-atomic-host/7/x86_64/standard", : "b672bf8a457cb28e003dee20c53749636ef5fce3e4743afe4aaad269d3aaa62a" }
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 193, in main
    return _auto_manage_db(options)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 256, in _auto_manage_db
    migrate_database(options)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 142, in migrate_database
    ensure_database_indexes()
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 179, in ensure_database_indexes
    model_class.ensure_indexes()
  File "/usr/lib/python2.7/site-packages/mongoengine/document.py", line 738, in ensure_indexes
    collection = cls._get_collection()
  File "/usr/lib/python2.7/site-packages/mongoengine/document.py", line 210, in _get_collection
    cls.ensure_indexes()
  File "/usr/lib/python2.7/site-packages/mongoengine/document.py", line 766, in ensure_indexes
    collection.create_index(fields, background=background, **opts)
  File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 1380, in create_index
    self.__create_index(keys, kwargs)
  File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 1290, in __create_index
    sock_info, cmd, read_preference=ReadPreference.PRIMARY)
  File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 205, in _command
    read_concern=read_concern)
  File "/usr/lib64/python2.7/site-packages/pymongo/pool.py", line 211, in command
    read_concern)
  File "/usr/lib64/python2.7/site-packages/pymongo/network.py", line 100, in command
    helpers._check_command_response(response_doc, msg, allowable_errors)
  File "/usr/lib64/python2.7/site-packages/pymongo/helpers.py", line 189, in _check_command_response
    raise DuplicateKeyError(errmsg, code, response)
DuplicateKeyError: E11000 duplicate key error index: pulp_database.units_ostree.$remote_id_1_branch_1_commit_1  dup key: { : "174b6356ebe246768ac1600709cfefc6589a43503397a9e7589111d4da9188f6", : "rhel-atomic-host/7/x86_64/standard", : "b672bf8a457cb28e003dee20c53749636ef5fce3e4743afe4aaad269d3aaa62a" }


Expected results:
no such fail


Additional info:
- as written, removing duplicates from units_ostree and references to them from repo_content_units collection, pulp-manage-db fails the same way
- foreman-debug to be attached
Comment 3 Pavel Moravec 2016-08-17 19:45:23 UTC 
Update: what I understood, the upgrade itself does not cause the duplicates. Rather they exist before it. But after the upgrade, new unique key-index is added to units_ostree collection.

Steps to resolve/workaround:
- grab mongo collections units_ostree, repo_content_units and lazy_content_catalog (or some other that can refer to units_ostree, if there is)
- identify the duplicates in units_ostree - mark one representant in each set of duplicates
- in other collected tables, update them to replace duplicates by their representants. Such that no reference to the dups exists.
- remove the dups from units_ostree
Comment 5 Pavel Moravec 2016-08-18 20:32:16 UTC 
This was most probably caused by running pulp-manage-db with pulp workers running. If some worker adds an OSTree unit that time, when uniqueness constrains may not be enforced, the unique index can get broken.

Closing as not a bug, created KCS 2534911 how to workaround / fix such dups created.
Note You need to log in before you can comment on or make changes to this bug.