Created attachment 1305438 [details] foreman production.log (see 2017-07-27 13:56) Description of problem: When adding more than one server to Satellite in quick succession there are DuplicateKeyError issues Version-Release number of selected component (if applicable): RHEL7.3 (fully updated), Satellite 6.2.10 (fully updated), vanilla configuration. How reproducible: Seemingly every time I add more than one server at a time. But it _might_ have happened when I added one server on its own. In this instance it was adding three servers in quick succession. Steps to Reproduce: 1. Run Ansible playbook over three hosts to use the subscription-manager module 2. Hosts apparently register correctly, but there's a stopped, error task Actual results: DuplicateKeyErrors Expected results: No issues Additional info: See attached /var/log/messages and /var/log/foreman/production.log files. Issues begin at Jul 27 13:56:22 (or just before).
Created attachment 1305439 [details] /var/log/messages (see 2017-07-27 13:56)
This error apparently appeared again today at 10:40: Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19163-81920) Task pulp.server.managers.consumer.applicability.regenerate_applicability_for_consumers[b3b65e2e-63a5-4db0-820d-9d784f396a9e] raised unexpected: DuplicateKeyError(u'insertDocument :: caused by :: 11000 E11000 duplicate key error index: pulp_database.repo_profile_applicability.$profile_hash_-1_repo_id_-1 dup key: { : "c49a8ef5a2d53da47fc85a26bc64c2da9120a96f1e3118fa459275bd5c50d1b2", : "UIS-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_7_Server_RPMs_x86_64_7Server" }',) Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19163-81920) Traceback (most recent call last): Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19163-81920) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19163-81920) R = retval = fun(*args, **kwargs) Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19163-81920) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 473, in __call__ Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19163-81920) return super(Task, self).__call__(*args, **kwargs) Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19163-81920) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in __call__ Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19163-81920) return super(PulpTask, self).__call__(*args, **kwargs) Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19163-81920) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__ Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19163-81920) return self.run(*args, **kwargs) Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19163-81920) File "/usr/lib/python2.7/site-packages/pulp/server/managers/consumer/applicability.py", line 107, in regenerate_applicability_for_consumers Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19163-81920) manager.regenerate_applicability(profile_hash, content_type, profile_id, repo_id) Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19163-81920) File "/usr/lib/python2.7/site-packages/pulp/server/managers/consumer/applicability.py", line 272, in regenerate_applicability Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19163-81920) applicability) Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19163-81920) File "/usr/lib/python2.7/site-packages/pulp/server/managers/consumer/applicability.py", line 384, in create Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19163-81920) applicability.save() Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19163-81920) File "/usr/lib/python2.7/site-packages/pulp/server/db/model/consumer.py", line 234, in save Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19163-81920) self._id = self.get_collection().insert(new_document) Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19163-81920) File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 2200, in insert Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19163-81920) check_keys, manipulate, write_concern) Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19163-81920) File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 530, in _insert Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19163-81920) check_keys, manipulate, write_concern, op_id, bypass_doc_val) Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19163-81920) File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 513, in _insert_one Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19163-81920) _check_write_command_response([(0, result)]) Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19163-81920) File "/usr/lib64/python2.7/site-packages/pymongo/helpers.py", line 300, in _check_write_command_response Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19163-81920) raise DuplicateKeyError(error.get("errmsg"), 11000, error) Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19163-81920) DuplicateKeyError: insertDocument :: caused by :: 11000 E11000 duplicate key error index: pulp_database.repo_profile_applicability.$profile_hash_-1_repo_id_-1 dup key: { : "c49a8ef5a2d53da47fc85a26bc64c2da9120a96f1e3118fa459275bd5c50d1b2", : "UIS-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_7_Server_RPMs_x86_64_7Server" } Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19166-49504) Task pulp.server.managers.consumer.applicability.regenerate_applicability_for_consumers[7b677453-d300-4b6c-9db1-0fc23a01377b] raised unexpected: DuplicateKeyError(u'insertDocument :: caused by :: 11000 E11000 duplicate key error index: pulp_database.repo_profile_applicability.$profile_hash_-1_repo_id_-1 dup key: { : "c49a8ef5a2d53da47fc85a26bc64c2da9120a96f1e3118fa459275bd5c50d1b2", : "UIS-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_7_Server_RPMs_x86_64_7Server" }',) Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19166-49504) Traceback (most recent call last): Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19166-49504) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19166-49504) R = retval = fun(*args, **kwargs) Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19166-49504) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 473, in __call__ Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19166-49504) return super(Task, self).__call__(*args, **kwargs) Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19166-49504) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in __call__ Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19166-49504) return super(PulpTask, self).__call__(*args, **kwargs) Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19166-49504) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__ Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19166-49504) return self.run(*args, **kwargs) Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19166-49504) File "/usr/lib/python2.7/site-packages/pulp/server/managers/consumer/applicability.py", line 107, in regenerate_applicability_for_consumers Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19166-49504) manager.regenerate_applicability(profile_hash, content_type, profile_id, repo_id) Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19166-49504) File "/usr/lib/python2.7/site-packages/pulp/server/managers/consumer/applicability.py", line 272, in regenerate_applicability Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19166-49504) applicability) Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19166-49504) File "/usr/lib/python2.7/site-packages/pulp/server/managers/consumer/applicability.py", line 384, in create Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19166-49504) applicability.save() Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19166-49504) File "/usr/lib/python2.7/site-packages/pulp/server/db/model/consumer.py", line 234, in save Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19166-49504) self._id = self.get_collection().insert(new_document) Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19166-49504) File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 2200, in insert Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19166-49504) check_keys, manipulate, write_concern) Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19166-49504) File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 530, in _insert Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19166-49504) check_keys, manipulate, write_concern, op_id, bypass_doc_val) Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19166-49504) File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 513, in _insert_one Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19166-49504) _check_write_command_response([(0, result)]) Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19166-49504) File "/usr/lib64/python2.7/site-packages/pymongo/helpers.py", line 300, in _check_write_command_response Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19166-49504) raise DuplicateKeyError(error.get("errmsg"), 11000, error) Jul 28 10:40:24 satellite1 pulp: celery.worker.job:ERROR: (19166-49504) DuplicateKeyError: insertDocument :: caused by :: 11000 E11000 duplicate key error index: pulp_database.repo_profile_applicability.$profile_hash_-1_repo_id_-1 dup key: { : "c49a8ef5a2d53da47fc85a26bc64c2da9120a96f1e3118fa459275bd5c50d1b2", : "UIS-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_7_Server_RPMs_x86_64_7Server" } I have no explanation as to why. The same three servers were told to "yum install unison" at that point.
*** This bug has been marked as a duplicate of bug 1468022 ***