Bug 1367758
| Summary: | [Sat6.2 upgrade] pulp-manage-db creates duplicate records in units_ostree | ||
|---|---|---|---|
| Product: | Red Hat Satellite | Reporter: | Pavel Moravec <pmoravec> |
| Component: | Pulp | Assignee: | satellite6-bugs <satellite6-bugs> |
| Status: | CLOSED NOTABUG | QA Contact: | Katello QA List <katello-qa-list> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 6.2.0 | CC: | donf, sramacha |
| Target Milestone: | Unspecified | ||
| Target Release: | Unused | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-08-18 20:32:30 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Update: what I understood, the upgrade itself does not cause the duplicates. Rather they exist before it. But after the upgrade, new unique key-index is added to units_ostree collection. Steps to resolve/workaround: - grab mongo collections units_ostree, repo_content_units and lazy_content_catalog (or some other that can refer to units_ostree, if there is) - identify the duplicates in units_ostree - mark one representant in each set of duplicates - in other collected tables, update them to replace duplicates by their representants. Such that no reference to the dups exists. - remove the dups from units_ostree This was most probably caused by running pulp-manage-db with pulp workers running. If some worker adds an OSTree unit that time, when uniqueness constrains may not be enforced, the unique index can get broken. Closing as not a bug, created KCS 2534911 how to workaround / fix such dups created. |
Description of problem: Upgrade of Satellite to 6.2 fails on step pulp-manage-db with backtrace suggesting units_ostree collection violates unique key/index. Manually removing duplicates (and references to them) and re-running just pulp-manage-db fails the same way. Version-Release number of selected component (if applicable): Sat 6.1.9 -> 6.2 upgrade pulp-server 2.6.0.21-1 -> 2.8.3.3-1 upgrade How reproducible: ??? I guess 100% Steps to Reproduce: n.a. at the moment, I guess it is enough to sync an ostree unit in 6.1, put to a repo, publish the repo, and upgrade to 6.2 Actual results: upgrade fails with: E11000 duplicate key error index: pulp_database.units_ostree.$remote_id_1_branch_1_commit_1 dup key: { : "174b6356ebe246768ac1600709cfefc6589a43503397a9e7589111d4da9188f6", : "rhel-atomic-host/7/x86_64/standard", : "b672bf8a457cb28e003dee20c53749636ef5fce3e4743afe4aaad269d3aaa62a" } Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 193, in main return _auto_manage_db(options) File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 256, in _auto_manage_db migrate_database(options) File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 142, in migrate_database ensure_database_indexes() File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 179, in ensure_database_indexes model_class.ensure_indexes() File "/usr/lib/python2.7/site-packages/mongoengine/document.py", line 738, in ensure_indexes collection = cls._get_collection() File "/usr/lib/python2.7/site-packages/mongoengine/document.py", line 210, in _get_collection cls.ensure_indexes() File "/usr/lib/python2.7/site-packages/mongoengine/document.py", line 766, in ensure_indexes collection.create_index(fields, background=background, **opts) File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 1380, in create_index self.__create_index(keys, kwargs) File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 1290, in __create_index sock_info, cmd, read_preference=ReadPreference.PRIMARY) File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 205, in _command read_concern=read_concern) File "/usr/lib64/python2.7/site-packages/pymongo/pool.py", line 211, in command read_concern) File "/usr/lib64/python2.7/site-packages/pymongo/network.py", line 100, in command helpers._check_command_response(response_doc, msg, allowable_errors) File "/usr/lib64/python2.7/site-packages/pymongo/helpers.py", line 189, in _check_command_response raise DuplicateKeyError(errmsg, code, response) DuplicateKeyError: E11000 duplicate key error index: pulp_database.units_ostree.$remote_id_1_branch_1_commit_1 dup key: { : "174b6356ebe246768ac1600709cfefc6589a43503397a9e7589111d4da9188f6", : "rhel-atomic-host/7/x86_64/standard", : "b672bf8a457cb28e003dee20c53749636ef5fce3e4743afe4aaad269d3aaa62a" } Expected results: no such fail Additional info: - as written, removing duplicates from units_ostree and references to them from repo_content_units collection, pulp-manage-db fails the same way - foreman-debug to be attached