Description of problem: foreman-maintain failed to cancel or resume the task after the satellite upgrade. Version-Release number of selected component (if applicable): 6.10 snap11 How reproducible: always Steps to Reproduce: 1. Prepare the satellite's based version 6.9 for the upgrade. 2. Run the pulp migration 3. Pulp migration completes successfully. 4. Run the satellite upgrade, satellite upgrade completes successfully but the post-upgrade job gets failed at "Check for paused tasks: " stage gets failed. satellite-maintain upgrade run --target-version 6.10 ................................. ................................. Check whether all services are running using the ping call: [OK] -------------------------------------------------------------------------------- Check for paused tasks: [FAIL] There are currently 6 paused tasks in the system -------------------------------------------------------------------------------- There are multiple steps to proceed: 1) Resume paused tasks 2) Delete paused tasks 3) Investigate the tasks via UI Select step to continue, [n(next)] 1 Resume paused tasks: / Waiting 30 seconds for resumed tasks to start. [OK] Total tasks resumed: 6 -------------------------------------------------------------------------------- Rerunning the check after fix procedure Check for paused tasks: [FAIL] There are currently 6 paused tasks in the system -------------------------------------------------------------------------------- There are multiple steps to proceed: 1) Resume paused tasks 2) Delete paused tasks 3) Investigate the tasks via UI Select step to continue, [n(next)] Select step to continue, [n(next)] n Check to verify no empty CA cert requests exist: [OK] -------------------------------------------------------------------------------- Check whether system is self-registered or not: [OK] -------------------------------------------------------------------------------- Check if only installed assets are present on the system: \ Checking for presence of non-original assets... [OK] -------------------------------------------------------------------------------- Print pulp 2 removal instructions: ====================================================== Migration of content from Pulp 2 to Pulp3 is complete After verifying accessibility of content from clients, it is strongly recommend to run "foreman-maintain content remove-pulp2" This will remove Pulp 2, MongoDB, and all pulp2 content in /var/lib/pulp/content/ ====================================================== [OK] -------------------------------------------------------------------------------- Scenario [Checks after upgrading to Satellite 6.10] failed. The following steps ended up in failing state: [foreman-tasks-not-paused] Resolve the failed steps and rerun the command. In case the failures are false positives, use --whitelist="foreman-tasks-not-paused" Actual results: post-upgrade job gets failed due to unable to resume or cancel the paused task. Expected results: post-upgrade task should be completed successfully. Additional info:
Here's the error from the refresh distribution tasks: 2021-07-30T04:22:16 [E|bac|] undefined method `new' for nil:NilClass (NoMethodError) | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.29/app/services/katello/pulp3/repository.rb:90:in `instance_for_type' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.29/app/models/katello/repository.rb:188:in `backend_service' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.29/app/lib/actions/pulp3/repository/refresh_distribution.rb:27:in `invoke_external_task' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:84:in `initiate_external_action' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:19:in `run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/cancellable.rb:14:in `run' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.29/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in `block (3 levels) in execute_run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.29/app/lib/actions/middleware/execute_if_contents_changed.rb:5:in `run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call' I suspect it may be hitting some ostree/puppet repos?
Created redmine issue https://projects.theforeman.org/issues/33206 from this bug
One workaround for this is to remove any ostree or puppet repos prior to upgrade
Verified on Satellite 6.10, snap 14. Steps to Test: 1. Deploy Satellite 6.9.5, snap 2. 2. Create a new product. 3. Create a Puppet repository in the new product. 4. Upload a Puppet module to the repository. 5. Enable and synchronize the rhel-atomic-host-ostree repository. 6. Migrate from Pulp 2 to Pulp 3: # satellite-maintain prep-6.10-upgrade # satellite-maintain content prepare 7. Upgrade the Satellite from 6.9 to 6.10. Expected Results: The upgrade completes successfully with no errors on RefreshDistribution tasks. Actual Results: The upgrade completes successfully with no errors on RefreshDistribution tasks.
*** Bug 1995717 has been marked as a duplicate of this bug. ***
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Satellite 6.10 Release), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:4702