Bug 1367758 - [Sat6.2 upgrade] pulp-manage-db creates duplicate records in units_ostree
Summary: [Sat6.2 upgrade] pulp-manage-db creates duplicate records in units_ostree
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.2.0
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: Katello QA List
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-08-17 12:00 UTC by Pavel Moravec
Modified: 2019-12-16 06:23 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-08-18 20:32:30 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1374723 0 high CLOSED [Improvement] prevent running pulp-manage-db when pulp workers are up 2021-04-06 17:56:01 UTC
Red Hat Knowledge Base (Solution) 2534911 0 None None None 2016-08-18 20:32:15 UTC

Internal Links: 1374723

Description Pavel Moravec 2016-08-17 12:00:37 UTC
Description of problem:
Upgrade of Satellite to 6.2 fails on step pulp-manage-db with backtrace suggesting units_ostree collection violates unique key/index.

Manually removing duplicates (and references to them) and re-running just pulp-manage-db fails the same way.


Version-Release number of selected component (if applicable):
Sat 6.1.9 -> 6.2 upgrade
pulp-server 2.6.0.21-1 -> 2.8.3.3-1 upgrade


How reproducible:
??? I guess 100%


Steps to Reproduce:
n.a. at the moment, I guess it is enough to sync an ostree unit in 6.1, put to a repo, publish the repo, and upgrade to 6.2


Actual results:
upgrade fails with:

E11000 duplicate key error index: pulp_database.units_ostree.$remote_id_1_branch_1_commit_1  dup key: { : "174b6356ebe246768ac1600709cfefc6589a43503397a9e7589111d4da9188f6", : "rhel-atomic-host/7/x86_64/standard", : "b672bf8a457cb28e003dee20c53749636ef5fce3e4743afe4aaad269d3aaa62a" }
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 193, in main
    return _auto_manage_db(options)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 256, in _auto_manage_db
    migrate_database(options)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 142, in migrate_database
    ensure_database_indexes()
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 179, in ensure_database_indexes
    model_class.ensure_indexes()
  File "/usr/lib/python2.7/site-packages/mongoengine/document.py", line 738, in ensure_indexes
    collection = cls._get_collection()
  File "/usr/lib/python2.7/site-packages/mongoengine/document.py", line 210, in _get_collection
    cls.ensure_indexes()
  File "/usr/lib/python2.7/site-packages/mongoengine/document.py", line 766, in ensure_indexes
    collection.create_index(fields, background=background, **opts)
  File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 1380, in create_index
    self.__create_index(keys, kwargs)
  File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 1290, in __create_index
    sock_info, cmd, read_preference=ReadPreference.PRIMARY)
  File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 205, in _command
    read_concern=read_concern)
  File "/usr/lib64/python2.7/site-packages/pymongo/pool.py", line 211, in command
    read_concern)
  File "/usr/lib64/python2.7/site-packages/pymongo/network.py", line 100, in command
    helpers._check_command_response(response_doc, msg, allowable_errors)
  File "/usr/lib64/python2.7/site-packages/pymongo/helpers.py", line 189, in _check_command_response
    raise DuplicateKeyError(errmsg, code, response)
DuplicateKeyError: E11000 duplicate key error index: pulp_database.units_ostree.$remote_id_1_branch_1_commit_1  dup key: { : "174b6356ebe246768ac1600709cfefc6589a43503397a9e7589111d4da9188f6", : "rhel-atomic-host/7/x86_64/standard", : "b672bf8a457cb28e003dee20c53749636ef5fce3e4743afe4aaad269d3aaa62a" }


Expected results:
no such fail


Additional info:
- as written, removing duplicates from units_ostree and references to them from repo_content_units collection, pulp-manage-db fails the same way
- foreman-debug to be attached

Comment 3 Pavel Moravec 2016-08-17 19:45:23 UTC
Update: what I understood, the upgrade itself does not cause the duplicates. Rather they exist before it. But after the upgrade, new unique key-index is added to units_ostree collection.

Steps to resolve/workaround:
- grab mongo collections units_ostree, repo_content_units and lazy_content_catalog (or some other that can refer to units_ostree, if there is)
- identify the duplicates in units_ostree - mark one representant in each set of duplicates
- in other collected tables, update them to replace duplicates by their representants. Such that no reference to the dups exists.
- remove the dups from units_ostree

Comment 5 Pavel Moravec 2016-08-18 20:32:16 UTC
This was most probably caused by running pulp-manage-db with pulp workers running. If some worker adds an OSTree unit that time, when uniqueness constrains may not be enforced, the unique index can get broken.

Closing as not a bug, created KCS 2534911 how to workaround / fix such dups created.


Note You need to log in before you can comment on or make changes to this bug.