Bug 1301335

Summary: Renaming organization, deleting&importing manifest causes subsequent CDN sync forbidden errors
Product: Red Hat Satellite Reporter: Pavel Moravec <pmoravec>
Component: Content ManagementAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED WONTFIX QA Contact: Katello QA List <katello-qa-list>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.2.0CC: bbuckingham, bkearney, james.oden
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-01 20:34:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Pavel Moravec 2016-01-24 10:48:14 UTC
Description of problem:
After renaming organization, doing some "get out, get in" safety routine in deleting manifest and reimporting it back, subsequent repository sync from CDN sometimes fails with yum metadata forbidden error.

Interestingly, refreshing the manifest does not seem to trigger the bug. Just delete & import the same again.

(but since the bz appears with some probability, I might not be (un)lucky enough when testing manifest refresh and repo sync)


Version-Release number of selected component (if applicable):
Sat 6.1.5


How reproducible:
50%


Steps to Reproduce:
1. Have Sat6 in use (with manifest imported, repos enabled and synced)
2. Rename organization
3. Delete manifest, import the same again
4. Sync repositories (already synced before the step 1) - the more the better, since the bug occurs with some probability, randomly for random repos in individual test cases


Actual results:
some repos sync fails with:

3: Actions::Pulp::Repository::Sync (skipped) [ 35.34s / 2.66s ]

Started at: 2016-01-24 10:32:32 UTC

Ended at: 2016-01-24 10:33:07 UTC

Real time: 35.34s

Execution time (excluding suspended state): 2.66s

Input:

---
pulp_id: Default_Organization-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_7_Server_RPMs_x86_64_7Server
task_id: 
remote_user: admin-9a78af3c
remote_cp_user: admin
locale: en

Output:

---
pulp_tasks:
- exception: 
  task_type: pulp.server.managers.repo.sync.sync
  _href: /pulp/api/v2/tasks/0eb2aa56-79f7-46a9-b5d9-7f0f8a40428b/
  task_id: 0eb2aa56-79f7-46a9-b5d9-7f0f8a40428b
  tags:
  - pulp:repository:Default_Organization-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_7_Server_RPMs_x86_64_7Server
  - pulp:action:sync
  finish_time: '2016-01-24T10:33:04Z'
  _ns: task_status
  start_time: '2016-01-24T10:33:03Z'
  traceback: ! "Traceback (most recent call last):\n  File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\",
    line 240, in trace_task\n    R = retval = fun(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\",
    line 328, in __call__\n    return super(Task, self).__call__(*args, **kwargs)\n
    \ File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 437, in
    __protected_call__\n    return self.run(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/pulp/server/managers/repo/sync.py\",
    line 114, in sync\n    raise PulpExecutionException(_('Importer indicated a failed
    response'))\nPulpExecutionException: Importer indicated a failed response\n"
  spawned_tasks: []
  progress_report:
    yum_importer:
      content:
        size_total: 0
        items_left: 0
        items_total: 0
        state: NOT_STARTED
        size_left: 0
        details:
          rpm_total: 0
          rpm_done: 0
          drpm_total: 0
          drpm_done: 0
        error_details: []
      comps:
        state: NOT_STARTED
      distribution:
        items_total: 0
        state: NOT_STARTED
        error_details: []
        items_left: 0
      errata:
        state: NOT_STARTED
      metadata:
        state: FAILED
        error: Forbidden
  queue: reserved_resource_worker-2.brq.redhat.com.dq
  state: error
  worker_name: reserved_resource_worker-2.brq.redhat.com
  result: 
  error:
    code: PLP0000
    data: {}
    description: Importer indicated a failed response
    sub_errors: []
  _id:
    $oid: 56a4a841c441a805a18c3113
  id: 56a4a841c44bb25302c1dd0d
poll_attempts:
  total: 16
  failed: 0

Error:

Katello::Errors::PulpError

PLP0000: Importer indicated a failed response

---
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.77/app/lib/actions/pulp/abstract_async_task.rb:83:in
  `block in done?'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.77/app/lib/actions/pulp/abstract_async_task.rb:81:in
  `each'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.77/app/lib/actions/pulp/abstract_async_task.rb:81:in
  `done?'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action/polling.rb:28:in
  `run'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action/cancellable.rb:9:in
  `run'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.77/app/lib/actions/pulp/abstract_async_task.rb:57:in
  `run'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:487:in
  `block (3 levels) in execute_run'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware.rb:16:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware.rb:25:in
  `run'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:22:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware.rb:16:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.77/app/lib/actions/middleware/remote_action.rb:27:in
  `block in run'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.77/app/lib/actions/middleware/remote_action.rb:57:in
  `block (2 levels) in as_remote_user'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.77/app/lib/katello/util/thread_session.rb:84:in
  `pulp_config'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.77/app/lib/actions/middleware/remote_action.rb:43:in
  `as_pulp_user'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.77/app/lib/actions/middleware/remote_action.rb:56:in
  `block in as_remote_user'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.77/app/lib/katello/util/thread_session.rb:91:in
  `cp_config'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.77/app/lib/actions/middleware/remote_action.rb:38:in
  `as_cp_user'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.77/app/lib/actions/middleware/remote_action.rb:55:in
  `as_remote_user'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.77/app/lib/actions/middleware/remote_action.rb:27:in
  `run'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:22:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware.rb:16:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action/progress.rb:30:in
  `with_progress_calculation'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action/progress.rb:16:in
  `run'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:22:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware.rb:16:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.77/app/lib/actions/middleware/keep_locale.rb:23:in
  `block in run'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.77/app/lib/actions/middleware/keep_locale.rb:34:in
  `with_locale'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.77/app/lib/actions/middleware/keep_locale.rb:23:in
  `run'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:22:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/world.rb:30:in
  `execute'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:486:in
  `block (2 levels) in execute_run'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:485:in
  `catch'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:485:in
  `block in execute_run'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in
  `block in with_error_handling'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in
  `catch'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in
  `with_error_handling'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:480:in
  `execute_run'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:262:in
  `execute'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in
  `block (2 levels) in execute'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract.rb:155:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract.rb:155:in
  `with_meta_calculation'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in
  `block in execute'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in
  `open_action'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in
  `execute'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/executors/parallel/worker.rb:20:in
  `block in on_message'
- /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:859:in
  `block in assigns'
- /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in
  `tap'
- /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in
  `assigns'
- /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:138:in
  `match_value'
- /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:116:in
  `block in match'
- /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in
  `each'
- /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in
  `match'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/executors/parallel/worker.rb:17:in
  `on_message'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:82:in
  `on_envelope'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:72:in
  `receive'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in
  `block (2 levels) in run'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in
  `loop'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in
  `block in run'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in
  `catch'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in
  `run'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:13:in
  `block in initialize'
- /opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in
  `block in create_with_logging_context'


Expected results:
no such failure


Additional info:

Comment 2 Pavel Moravec 2016-01-24 15:07:21 UTC
(automated) testing scenario variant with manifest refresh, it never hit the pulp error. So the error appears _only_ when deleting and importing the manifest.

Comment 3 James Olin Oden 2016-03-09 21:01:37 UTC
I don't know if this is related, but I recently deleted a manifest and re-imported but set a different CDN URL.   When I did this, the UI showed the new URL, but "hammer repository info" showed the old one.  This happened while I was going through the ROLE class on Satellite 6.1 and I made a mistake by not editing the CDN entry before import, and thus needed to re-import (well at least I thought I did).

Comment 4 James Olin Oden 2016-03-09 21:02:38 UTC
BTW, I was logged in just now with my personal email.   I now work at Red Hat and my email is joden.

Comment 5 Bryan Kearney 2016-07-26 15:25:24 UTC
Moving 6.2 bugs out to sat-backlog.

Comment 6 Bryan Kearney 2016-07-26 15:36:59 UTC
Moving 6.2 bugs out to sat-backlog.

Comment 8 Bryan Kearney 2017-08-01 20:34:32 UTC
Thank you for your interest in Satellite 6. We have evaluated this request, and we do not expect this to be implemented in product in the foreseeable future. We are therefore closing this out as WONTFIX. If you have any concerns about this, please feel free to contact Rich Jerrido or Bryan Kearney. Thank you.