1745835 – Full Capsule sync doesn't fix the broken repository metadata and results in ERROR: "undefined method `[]' for nil:NilClass"

Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1745835 - Full Capsule sync doesn't fix the broken repository metadata and results in ERROR: "undefined method `[]' for nil:NilClass"

Summary: Full Capsule sync doesn't fix the broken repository metadata and results in E...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Satellite
Classification:	Red Hat
Component:	Capsule - Content
Sub Component:
Version:	6.5.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	high
Target Milestone:	6.7.0
Assignee:	Samir Jha
QA Contact:	Lukas Pramuk
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2019-08-27 04:23 UTC by Hao Chang Yu
Modified:	2023-09-07 20:28 UTC (History)
CC List:	14 users (show)
Fixed In Version:	rubygem-katello-3.12.0.21-1
Doc Type:	Known Issue
Doc Text:	Full Capsule synchronization fails and does not fully publish new metadata.
Clone Of:
Clones:	1769893 (view as bug list)
Environment:
Last Closed:	2020-04-14 13:25:28 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Foreman Issue Tracker	27776	Normal	Closed	Full Capsule sync doesn't fix the broken repository metadata.	2021-02-19 10:15:33 UTC
Foreman Issue Tracker	28046	Normal	Closed	Smart Proxy sync fails with undefined method `[]' for nil:NilClass (NoMethodError)	2021-02-19 10:15:34 UTC
Red Hat Product Errata	RHSA-2020:1454	None	None	None	2020-04-14 13:25:45 UTC

Description Hao Chang Yu 2019-08-27 04:23:49 UTC

Description of problem:
When doing a force full Capsule sync, Satellite incorrectly running the regenerate metadata tasks ("Actions::Pulp::Repository::DistributorPublish") on itself instead of running it on the Capsule.

See the below output:

274: Actions::Pulp::Consumer::SyncCapsule (success) [ 4113.54s / 803.05s ]
Started at: 2019-08-27 02:41:15 UTC
Ended at: 2019-08-27 03:49:49 UTC
Real time: 4113.54s
Execution time (excluding suspended state): 803.05s

Input:
---
capsule_id: 2
repo_pulp_id: 80bd2431-8c57-45fe-a290-de4f311f6545
sync_options:
  remove_missing: true
  force_full: true

Output:
---
pulp_tasks:
- exception: 
  task_type: pulp.server.managers.repo.sync.sync
  _href: "/pulp/api/v2/tasks/1cb5b017-950e-4356-bbce-d9ed3f53e1a9/"
  task_id: 1cb5b017-950e-4356-bbce-d9ed3f53e1a9
  tags:
  - pulp:repository:80bd2431-8c57-45fe-a290-de4f311f6545
  - pulp:action:sync
  finish_time: '2019-08-27T02:42:03Z'
  _ns: task_status
  start_time: '2019-08-27T02:41:05Z'
  traceback: 
  spawned_tasks:
  - _href: "/pulp/api/v2/tasks/81c33bcf-7ea7-4e3d-88d2-b9109fe8840a/"
    task_id: 81c33bcf-7ea7-4e3d-88d2-b9109fe8840a
  progress_report:
    yum_importer:
    ...
  queue: reserved_resource_worker-0.com.dq2    <====
  state: finished
  worker_name: reserved_resource_worker-0.com  <==== Sync task was running in the Capsule which is correct


Input:
---
pulp_id: 80bd2431-8c57-45fe-a290-de4f311f6545
distributor_type_id: yum_distributor
source_pulp_id: 
override_config:
  force_full: true
dependency: 
matching_content: 
...

Output:
---
pulp_tasks:
- exception: 
  task_type: pulp.server.managers.repo.publish.publish
  _href: "/pulp/api/v2/tasks/539f81b0-529a-412d-a0dc-3bcc9b6a9831/"
  task_id: 539f81b0-529a-412d-a0dc-3bcc9b6a9831
  tags:
  - pulp:repository:80bd2431-8c57-45fe-a290-de4f311f6545
  - pulp:action:publish
  finish_time: '2019-08-27T03:50:07Z'
  _ns: task_status
  start_time: '2019-08-27T03:50:05Z'
  traceback: 
  spawned_tasks: []
  progress_report:
  ...
  queue: reserved_resource_worker-0.com.dq2
  state: finished
  worker_name: reserved_resource_worker-0.com  <====== Regenerate metadata was running in the Satellite.
...


Version-Release number of selected component (if applicable):
Satellite 6.4 and 6.5

How reproducible:
always


Steps to Reproduce:
1. Perform full Capsule sync from the Satellite web ui.

Comment 6 Samir Jha 2019-09-03 23:15:42 UTC

Created redmine issue http://projects.theforeman.org/issues/27776 from this bug

Comment 7 Bryan Kearney 2019-09-09 22:01:15 UTC

Moving this bug to POST for triage into Satellite 6 since the upstream issue https://projects.theforeman.org/issues/27776 has been resolved.

Comment 13 Lukas Pramuk 2019-10-10 12:21:01 UTC

FailedQA.

@Satellite 6.6.0 Snap22
tfm-rubygem-katello-3.12.0.27-1.el7sat.noarch

Full Capsule sync now ends with the error "undefined method `[]' for nil:NilClass" shown per each repo being synced.
Optimized sync works as expected. Not sure if the problem is caused by the fix.

Steps to reproduce the new issue:

1) Create a CV and assign a repo to it
2) Publish and promote CV to a LFE
3) Assign a LFE to external capsule
4) Trigger Full Capsule sync

 Label: Actions::Katello::CapsuleContent::Sync
Status: stopped
Result: warning 

_____/Run\__________
 3: Actions::Pulp::Repository::RefreshNeeded (success) [ 0.06s / 0.06s ]
 6: Actions::Pulp::Consumer::SyncCapsule (success) [ 11.95s / 0.29s ]
10: Actions::Pulp::Repository::DistributorPublish (skipped) [ 0.05s / 0.03s ]

Comment 14 Lukas Pramuk 2019-10-10 12:26:14 UTC

Actions::Pulp::Repository::DistributorPublish failed with the error

Started at: 2019-10-09 16:57:38 UTC
Ended at: 2019-10-09 16:57:38 UTC
Real time: 0.05s
Execution time (excluding suspended state): 0.03s

Input:
---
smart_proxy_id: 4
options:
  force: true
  source_repository:
    id: 45
    name: Capsule_6_6_RHEL7
    url: http://<SAT_FQDN>/pulp/repos/Default_Organization/content_views/Capsule_6_6/5.0/custom/Capsule/Capsule_6_6_RHEL7/
  matching_content: 
  dependency: 
  repository_creation: false
dependency: 
repository_id: 34
remote_user: admin
remote_cp_user: admin
current_request_id: 
current_timezone: Europe/Prague
current_user_id: 4
current_organization_id: 1
current_location_id: 

Output:
--- {}

Error:
NoMethodError
undefined method `[]' for nil:NilClass
---
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.12.0.27/app/services/katello/pulp/repository.rb:168:in
  `lookup_distributor_id'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.12.0.27/app/services/katello/pulp/repository/yum.rb:57:in
  `distributors_to_publish'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.12.0.27/app/services/katello/pulp/repository.rb:160:in
  `distributor_publish'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.12.0.27/app/lib/actions/pulp/repository/distributor_publish.rb:24:in
  `invoke_external_task'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/action/polling.rb:83:in
  `initiate_external_action'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/action/polling.rb:18:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/action/cancellable.rb:13:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.12.0.27/app/lib/actions/pulp/abstract_async_task.rb:45:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/action.rb:538:in
  `block (3 levels) in execute_run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:26:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware.rb:18:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.12.0.27/app/lib/actions/middleware/skip_if_matching_content.rb:5:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:22:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:26:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware.rb:18:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware.rb:31:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:22:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware/stack.rb:26:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/middleware.rb:18:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.12.0.27/app/lib/actions/middleware/remote_action.rb:16:in
  `block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.12.0.27/app/lib/actions/middleware/remote_action.rb:40:in
  `block in as_remote_user'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.12.0.27/app/models/katello/concerns/user_extensions.rb:21:in
  `cp_config'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.12.0.27/app/lib/actions/middleware/remote_action.rb:27:in
  `as_cp_user'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.12.0.27/app/lib/actions/middleware/remote_action.rb:39:in
  `as_remote_user'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.12.0.27/app/lib/actions/middleware/remote_action.rb:16:in
  `run'"
...

Comment 19 Lukas Pramuk 2019-11-27 16:57:03 UTC

VERIFIED.

@satellite-6.7.0-4.beta.el7sat.noarch
tfm-rubygem-katello-3.14.0-0.4.rc1.el7sat.noarch

by the following manual reproducer:

1) Create a CV and assign a repo to it
2) Publish and promote CV to a LFE
3) Assign a LFE to external capsule
4) Trigger Full Capsule sync

 Label: Actions::Katello::CapsuleContent::Sync
Status: stopped
Result: success

_____/Run\__________
3: Actions::Pulp::Orchestration::Repository::RefreshRepos (success) [ 0.23s / 0.23s ]
7: Actions::Pulp::Consumer::SyncCapsule (success) [ 17.86s / 1.75s ]
10: Actions::Pulp::Repository::DistributorPublish (success) [ 1.88s / 0.26s ] 

>>> the action Actions::Pulp::Repository::DistributorPublish succeeded 

5) Check where Actions::Pulp::Repository::DistributorPublish was run

 10: Actions::Pulp::Repository::DistributorPublish (success) [ 1.88s / 0.26s ]

Queue: default
Started at: 2019-11-27 13:35:57 UTC
Ended at: 2019-11-27 13:35:59 UTC
Real time: 1.88s
Execution time (excluding suspended state): 0.26s

Input:
---
smart_proxy_id: 2
options:
  force: true
  source_repository:
    id: 8
    name: Red_Hat_Ansible_Engine_2_8_RPMs_for_Red_Hat_Enterprise_Linux_7_Server_x86_64
    url: https://satellite.example.com/pulp/repos/Default_Organization/content_views/Test_View/1.0/content/dist/rhel/server/7/7Server/x86_64/ansible/2.8/os/
  matching_content: false
  dependency: 
  repository_creation: false
dependency: 
repository_id: 19
remote_user: admin
remote_cp_user: admin
current_request_id: 
current_timezone: Europe/Prague
current_user_id: 4
current_organization_id: 1
current_location_id: 2

Output:
---
pulp_tasks:
- exception: 
  task_type: pulp.server.managers.repo.publish.publish
  _href: "/pulp/api/v2/tasks/2843b068-933f-4d33-86cd-d5cd28330695/"
  task_id: 2843b068-933f-4d33-86cd-d5cd28330695
  tags:
  - pulp:repository:1-Test_View-Dev-fbae0800-7fc7-4a64-ad91-c71f8e901d34
  - pulp:action:publish
  finish_time: '2019-11-27T13:35:59Z'
  _ns: task_status
  start_time: '2019-11-27T13:35:58Z'
  traceback: 
  spawned_tasks: []
  progress_report:
  ...

  queue: reserved_resource_worker-2.com.dq2
  state: finished
  worker_name: reserved_resource_worker-2.com
  result:
    result: success
  ...

>>> both queue: and worker_name: indicate that DistributorPublish is being performed on the capsule

Comment 22 errata-xmlrpc 2020-04-14 13:25:28 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:1454

Note You need to log in before you can comment on or make changes to this bug.