Bug 1745835 - Full Capsule sync doesn't fix the broken repository metadata and results in ERROR: "undefined method `[]' for nil:NilClass" [NEEDINFO]
Summary: Full Capsule sync doesn't fix the broken repository metadata and results in E...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Capsule - Content
Version: 6.5.0
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: 6.7.0
Assignee: Samir Jha
QA Contact: Lukas Pramuk
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-08-27 04:23 UTC by Hao Chang Yu
Modified: 2020-04-14 13:25 UTC (History)
13 users (show)

Fixed In Version: rubygem-katello-3.12.0.21-1
Doc Type: Known Issue
Doc Text:
Full Capsule synchronization fails and does not fully publish new metadata.
Clone Of:
: 1769893 (view as bug list)
Environment:
Last Closed: 2020-04-14 13:25:28 UTC
Target Upstream Version:
fperalta: needinfo? (sajha)


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Foreman Issue Tracker 27776 Normal Closed Full Capsule sync doesn't fix the broken repository metadata. 2020-11-05 20:05:59 UTC
Foreman Issue Tracker 28046 Normal Closed Smart Proxy sync fails with undefined method `[]' for nil:NilClass (NoMethodError) 2020-11-05 20:05:59 UTC
Red Hat Product Errata RHSA-2020:1454 None None None 2020-04-14 13:25:45 UTC

Description Hao Chang Yu 2019-08-27 04:23:49 UTC
Description of problem:
When doing a force full Capsule sync, Satellite incorrectly running the regenerate metadata tasks ("Actions::Pulp::Repository::DistributorPublish") on itself instead of running it on the Capsule.

See the below output:

274: Actions::Pulp::Consumer::SyncCapsule (success) [ 4113.54s / 803.05s ]
Started at: 2019-08-27 02:41:15 UTC
Ended at: 2019-08-27 03:49:49 UTC
Real time: 4113.54s
Execution time (excluding suspended state): 803.05s

Input:
---
capsule_id: 2
repo_pulp_id: 80bd2431-8c57-45fe-a290-de4f311f6545
sync_options:
  remove_missing: true
  force_full: true

Output:
---
pulp_tasks:
- exception: 
  task_type: pulp.server.managers.repo.sync.sync
  _href: "/pulp/api/v2/tasks/1cb5b017-950e-4356-bbce-d9ed3f53e1a9/"
  task_id: 1cb5b017-950e-4356-bbce-d9ed3f53e1a9
  tags:
  - pulp:repository:80bd2431-8c57-45fe-a290-de4f311f6545
  - pulp:action:sync
  finish_time: '2019-08-27T02:42:03Z'
  _ns: task_status
  start_time: '2019-08-27T02:41:05Z'
  traceback: 
  spawned_tasks:
  - _href: "/pulp/api/v2/tasks/81c33bcf-7ea7-4e3d-88d2-b9109fe8840a/"
    task_id: 81c33bcf-7ea7-4e3d-88d2-b9109fe8840a
  progress_report:
    yum_importer:
    ...
  queue: reserved_resource_worker-0@hao-capsule.example.com.dq2    <====
  state: finished
  worker_name: reserved_resource_worker-0@hao-capsule.example.com  <==== Sync task was running in the Capsule which is correct


Input:
---
pulp_id: 80bd2431-8c57-45fe-a290-de4f311f6545
distributor_type_id: yum_distributor
source_pulp_id: 
override_config:
  force_full: true
dependency: 
matching_content: 
...

Output:
---
pulp_tasks:
- exception: 
  task_type: pulp.server.managers.repo.publish.publish
  _href: "/pulp/api/v2/tasks/539f81b0-529a-412d-a0dc-3bcc9b6a9831/"
  task_id: 539f81b0-529a-412d-a0dc-3bcc9b6a9831
  tags:
  - pulp:repository:80bd2431-8c57-45fe-a290-de4f311f6545
  - pulp:action:publish
  finish_time: '2019-08-27T03:50:07Z'
  _ns: task_status
  start_time: '2019-08-27T03:50:05Z'
  traceback: 
  spawned_tasks: []
  progress_report:
  ...
  queue: reserved_resource_worker-0@hao-satellite.example.com.dq2
  state: finished
  worker_name: reserved_resource_worker-0@hao-satellite.example.com  <====== Regenerate metadata was running in the Satellite.
...


Version-Release number of selected component (if applicable):
Satellite 6.4 and 6.5

How reproducible:
always


Steps to Reproduce:
1. Perform full Capsule sync from the Satellite web ui.

Comment 6 Samir Jha 2019-09-03 23:15:42 UTC
Created redmine issue http://projects.theforeman.org/issues/27776 from this bug

Comment 7 Bryan Kearney 2019-09-09 22:01:15 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue https://projects.theforeman.org/issues/27776 has been resolved.

Comment 13 Lukas Pramuk 2019-10-10 12:21:01 UTC
FailedQA.

@Satellite 6.6.0 Snap22
tfm-rubygem-katello-3.12.0.27-1.el7sat.noarch

Full Capsule sync now ends with the error "undefined method `[]' for nil:NilClass" shown per each repo being synced.
Optimized sync works as expected. Not sure if the problem is caused by the fix.

Steps to reproduce the new issue:

1) Create a CV and assign a repo to it
2) Publish and promote CV to a LFE
3) Assign a LFE to external capsule
4) Trigger Full Capsule sync

 Label: Actions::Katello::CapsuleContent::Sync
Status: stopped
Result: warning 

_____/Run\__________
 3: Actions::Pulp::Repository::RefreshNeeded (success) [ 0.06s / 0.06s ]
 6: Actions::Pulp::Consumer::SyncCapsule (success) [ 11.95s / 0.29s ]
10: Actions::Pulp::Repository::DistributorPublish (skipped) [ 0.05s / 0.03s ]

Comment 14 Lukas Pramuk 2019-10-10 12:26:14 UTC
Actions::Pulp::Repository::DistributorPublish failed with the error

Started at: 2019-10-09 16:57:38 UTC
Ended at: 2019-10-09 16:57:38 UTC
Real time: 0.05s
Execution time (excluding suspended state): 0.03s

Input:
---
smart_proxy_id: 4
options:
  force: true
  source_repository:
    id: 45
    name: Capsule_6_6_RHEL7
    url: http://<SAT_FQDN>/pulp/repos/Default_Organization/content_views/Capsule_6_6/5.0/custom/Capsule/Capsule_6_6_RHEL7/
  matching_content: 
  dependency: 
  repository_creation: false
dependency: 
repository_id: 34
remote_user: admin
remote_cp_user: admin
current_request_id: 
current_timezone: Europe/Prague
current_user_id: 4
current_organization_id: 1
current_location_id: 

Output:
--- {}

Error:
NoMethodError
undefined method `[]' for nil:NilClass
---
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.12.0.27/app/services/katello/pulp/repository.rb:168:in
  `lookup_distributor_id'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.12.0.27/app/services/katello/pulp/repository/yum.rb:57:in
  `distributors_to_publish'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.12.0.27/app/services/katello/pulp/repository.rb:160:in
  `distributor_publish'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.12.0.27/app/lib/actions/pulp/repository/distributor_publish.rb:24:in
  `invoke_external_task'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/action/polling.rb:83:in
  `initiate_external_action'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/action/polling.rb:18:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/action/cancellable.rb:13:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.12.0.27/app/lib/actions/pulp/abstract_async_task.rb:45:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/action.rb:538:in
  `block (3 levels) in execute_run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:26:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware.rb:18:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.12.0.27/app/lib/actions/middleware/skip_if_matching_content.rb:5:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:22:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:26:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware.rb:18:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware.rb:31:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:22:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:26:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware.rb:18:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.12.0.27/app/lib/actions/middleware/remote_action.rb:16:in
  `block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.12.0.27/app/lib/actions/middleware/remote_action.rb:40:in
  `block in as_remote_user'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.12.0.27/app/models/katello/concerns/user_extensions.rb:21:in
  `cp_config'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.12.0.27/app/lib/actions/middleware/remote_action.rb:27:in
  `as_cp_user'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.12.0.27/app/lib/actions/middleware/remote_action.rb:39:in
  `as_remote_user'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.12.0.27/app/lib/actions/middleware/remote_action.rb:16:in
  `run'"
...

Comment 19 Lukas Pramuk 2019-11-27 16:57:03 UTC
VERIFIED.

@satellite-6.7.0-4.beta.el7sat.noarch
tfm-rubygem-katello-3.14.0-0.4.rc1.el7sat.noarch

by the following manual reproducer:

1) Create a CV and assign a repo to it
2) Publish and promote CV to a LFE
3) Assign a LFE to external capsule
4) Trigger Full Capsule sync

 Label: Actions::Katello::CapsuleContent::Sync
Status: stopped
Result: success

_____/Run\__________
3: Actions::Pulp::Orchestration::Repository::RefreshRepos (success) [ 0.23s / 0.23s ]
7: Actions::Pulp::Consumer::SyncCapsule (success) [ 17.86s / 1.75s ]
10: Actions::Pulp::Repository::DistributorPublish (success) [ 1.88s / 0.26s ] 

>>> the action Actions::Pulp::Repository::DistributorPublish succeeded 

5) Check where Actions::Pulp::Repository::DistributorPublish was run

 10: Actions::Pulp::Repository::DistributorPublish (success) [ 1.88s / 0.26s ]

Queue: default
Started at: 2019-11-27 13:35:57 UTC
Ended at: 2019-11-27 13:35:59 UTC
Real time: 1.88s
Execution time (excluding suspended state): 0.26s

Input:
---
smart_proxy_id: 2
options:
  force: true
  source_repository:
    id: 8
    name: Red_Hat_Ansible_Engine_2_8_RPMs_for_Red_Hat_Enterprise_Linux_7_Server_x86_64
    url: https://satellite.example.com/pulp/repos/Default_Organization/content_views/Test_View/1.0/content/dist/rhel/server/7/7Server/x86_64/ansible/2.8/os/
  matching_content: false
  dependency: 
  repository_creation: false
dependency: 
repository_id: 19
remote_user: admin
remote_cp_user: admin
current_request_id: 
current_timezone: Europe/Prague
current_user_id: 4
current_organization_id: 1
current_location_id: 2

Output:
---
pulp_tasks:
- exception: 
  task_type: pulp.server.managers.repo.publish.publish
  _href: "/pulp/api/v2/tasks/2843b068-933f-4d33-86cd-d5cd28330695/"
  task_id: 2843b068-933f-4d33-86cd-d5cd28330695
  tags:
  - pulp:repository:1-Test_View-Dev-fbae0800-7fc7-4a64-ad91-c71f8e901d34
  - pulp:action:publish
  finish_time: '2019-11-27T13:35:59Z'
  _ns: task_status
  start_time: '2019-11-27T13:35:58Z'
  traceback: 
  spawned_tasks: []
  progress_report:
  ...

  queue: reserved_resource_worker-2@capsule.example.com.dq2
  state: finished
  worker_name: reserved_resource_worker-2@capsule.example.com
  result:
    result: success
  ...

>>> both queue: and worker_name: indicate that DistributorPublish is being performed on the capsule

Comment 22 errata-xmlrpc 2020-04-14 13:25:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:1454


Note You need to log in before you can comment on or make changes to this bug.