Bug 1311821

I saw this when updating a Red Hat repo. Here's the output from /var/log/messages:

Feb 25 08:16:33 katello-centos7-nightly2 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata from https://cdn.redhat.com/content/dist/rhel/server/5/5Server/x86_64/rhn-tools/os/.
Feb 25 08:16:33 katello-centos7-nightly2 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com
Feb 25 08:16:33 katello-centos7-nightly2 pulp: nectar.downloaders.threaded:WARNING: Connection Error - https://cdn.redhat.com/content/dist/rhel/server/5/5Server/x86_64/rhn-tools/os/repodata/repomd.xml could not be reached.
Feb 25 08:16:34 katello-centos7-nightly2 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com
Feb 25 08:16:34 katello-centos7-nightly2 pulp: nectar.downloaders.threaded:WARNING: Connection Error - https://cdn.redhat.com/content/dist/rhel/server/5/5Server/x86_64/rhn-tools/os could not be reached.
Feb 25 08:16:34 katello-centos7-nightly2 pulp: pulp.server.async.tasks:INFO: Task failed : [d6ead658-2524-4207-9364-f1d92a3ddd37] : Error retrieving metadata: Not found

I'm attaching pulp-admin outputs from before the update (original_repo_details) and after (new_repo_details).

The SSL info is changing so I am pretty sure this is related to #1311678.

Confirmed that deleting a manifest also sets ssl_ca_cert and ssl_client_cert to nil for a repo. Traced the cause to this bit of code:

https://git.io/v24F1

Those values are all nil.

Upstream bug assigned to jsherril

Upstream bug component is Content Management

I am seeing this issue even without a manifest that was deleted and re-imported.

---
pulp_tasks:
- exception: 
  task_type: pulp.server.managers.repo.sync.sync
  _href: "/pulp/api/v2/tasks/2dd91132-0a38-4d55-b22e-5de50e34ce7c/"
  task_id: 2dd91132-0a38-4d55-b22e-5de50e34ce7c
  tags:
  - pulp:repository:Default_Organization-Red_Hat_Satellite-Red_Hat_Satellite_6_1_for_RHEL_7_Server_RPMs_x86_64
  - pulp:action:sync
  finish_time: '2016-03-03T15:13:25Z'
  _ns: task_status
  start_time: '2016-03-03T15:13:24Z'
  traceback: |
    Traceback (most recent call last):
      File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
        R = retval = fun(*args, **kwargs)
      File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 473, in __call__
        return super(Task, self).__call__(*args, **kwargs)
      File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in __call__
        return super(PulpTask, self).__call__(*args, **kwargs)
      File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
        return self.run(*args, **kwargs)
      File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 757, in sync
        sync_report = sync_repo(transfer_repo, conduit, call_config)
      File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 658, in wrap_f
        return f(*args, **kwargs)
      File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/importer.py", line 81, in sync_repo
        report = self._current_sync.run()
      File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 199, in run
        raise PulpCodedException(error_code=error_codes.RPM1004, reason='Not found')
    PulpCodedException: Error retrieving metadata: Not found
  spawned_tasks: []
  progress_report:
    yum_importer:
      content:
        items_total: 0
        state: NOT_STARTED
        error_details: []
        details:
          rpm_total: 0
          rpm_done: 0
          drpm_total: 0
          drpm_done: 0
        size_total: 0
        size_left: 0
        items_left: 0
      comps:
        state: NOT_STARTED
      purge_duplicates:
        state: NOT_STARTED
      distribution:
        items_total: 0
        state: NOT_STARTED
        error_details: []
        items_left: 0
      errata:
        state: NOT_STARTED
      metadata:
        state: NOT_STARTED
  queue: reserved_resource_worker-3.lab.eng.bos.redhat.com.dq
  state: error
  worker_name: reserved_resource_worker-3.lab.eng.bos.redhat.com
  result: 
  error:
    code: RPM1004
    data:
      reason: Not found
    description: 'Error retrieving metadata: Not found'
    sub_errors: []
  _id:
    "$oid": 56d85493809b3e6d244b1288
  id: 56d85493809b3e6d244b1288
contents_changed: true
poll_attempts:
  total: 3
  failed: 0

Moving to POST since upstream bug http://projects.theforeman.org/issues/13991 has been closed
-------------
Justin Sherrill
Applied in changeset commit:katello|d7ad5dc027e3a5f97c5f016df4a64df0baf95d53.

Just a note, not only seen after Manifest delete and re-import this issue is also seen with a fresh setup.

And I don't think that issue is connected only to RH repos as I cannot sync even custom yum repo (fake one for example - http://inecas.fedorapeople.org/fakerepos/zoo3/) from first attempt (not re-sync) from direct API calls:

TaskFailedError: Task 71c7c522-e57a-4c22-b9de-68f0b9bf3e6a did not succeed. Task information: {u'username': u'admin', u'ended_at': u'2016-03-09 13:54:49 UTC', u'output': {}, u'cli_example': None, u'label': u'Actions::Katello::Repository::Sync', u'state': u'stopped', u'result': u'warning', u'input': {u'product': {u'label': u'55b45413-52d7-4077-879b-84b85aebf713', u'id': 53, u'cp_id': u'1457531532781', u'name': u'\u0240\U000266f0\u756b\uc7e6\u4797\u1d58\U000249fc\U0002aa4f\U0002b633\u66b7\u9dce\u8694\u7e95\u5dc0\U00026a23\U00023140\U0001d71e\U000130ff\U0002aa05\u9525\uacdb\uc71d\u9d09\u9b2b\u3cc7\uba80'}, u'user_id': 3, u'repository': {u'label': u'13889420-db41-4365-83bd-9689fd314ccb', u'id': 58, u'name': u'\U00023afb\u4e9d\u627a\U00013348\uc3a5\U00023089\u674d\U0001311b\u99df\u3dcd\U00022cd3'}, u'locale': u'en', u'contents_changed': {u'step_id': 3, u'subkeys': [u'contents_changed'], u'action_id': 2, u'class': u'Dynflow::ExecutionPlan::OutputReference', u'execution_plan_id': u'5f4da46d-6827-4dee-b84a-c886a2a60999'}, u'sync_result': {u'step_id': 3, u'subkeys': [], u'action_id': 2, u'class': u'Dynflow::ExecutionPlan::OutputReference', u'execution_plan_id': u'5f4da46d-6827-4dee-b84a-c886a2a60999'}, u'current_user_id': 3, u'provider': {u'id': 81, u'name': u'Anonymous'}, u'organization': {u'label': u'5d596427-6685-45f7-9411-47702ee24f61', u'id': 57, u'name': u'\U00028505\U000201bd\U000280c0\u0400\U000207c9\U00024d11\U000207dc\ub97c\U00028a94\U00024826\U000218db\u567a\xce\u513b\u809f\U0002b2d2\U000203bf\U00027156\U00029338\u9501\uaede'}, u'services_checked': [u'pulp', u'pulp_auth'], u'id': 58}, u'progress': 1.0, u'started_at': u'2016-03-09 13:54:44 UTC', u'humanized': {u'action': u'Synchronize', u'input': [[u'repository', {u'text': u"repository '\U00023afb\u4e9d\u627a\U00013348\uc3a5\U00023089\u674d\U0001311b\u99df\u3dcd\U00022cd3'", u'link': None}], [u'product', {u'text': u"product '\u0240\U000266f0\u756b\uc7e6\u4797\u1d58\U000249fc\U0002aa4f\U0002b633\u66b7\u9dce\u8694\u7e95\u5dc0\U00026a23\U00023140\U0001d71e\U000130ff\U0002aa05\u9525\uacdb\uc71d\u9d09\u9b2b\u3cc7\uba80'", u'link': u'#/products/53/info'}], [u'organization', {u'text': u"organization '\U00028505\U000201bd\U000280c0\u0400\U000207c9\U00024d11\U000207dc\ub97c\U00028a94\U00024826\U000218db\u567a\xce\u513b\u809f\U0002b2d2\U000203bf\U00027156\U00029338\u9501\uaede'", u'link': u'/organizations/57/edit'}]], u'errors': [u'RPM1004: Error retrieving metadata: Not found'], u'output': u''}, u'id': u'71c7c522-e57a-4c22-b9de-68f0b9bf3e6a', u'pending': False}

So basically I cannot synchronize any repository that I want. There is a chance that something can work for fresh setup, but with time, any repos stopped to work.
Just want to mention that issue seems as more generic one and that testing should be more extensive

(In reply to Oleksandr Shtaier from comment #14)
> So basically I cannot synchronize any repository that I want. There is a
> chance that something can work for fresh setup, but with time, any repos
> stopped to work.
> Just want to mention that issue seems as more generic one and that testing
> should be more extensive

You might be hitting this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1316229 which is caused by using an authenticated HTTP proxy

Verified in Satellite 6.2 Beta Snap 4. Manifest deleting/re-importing or refreshing no longer causes issues with repo sync.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1501