Bug 1993844

Summary: The system appears to already be using pulp3 with all content migrated.
Product: Red Hat Satellite Reporter: Kenny Tordeurs <ktordeur>
Component: Satellite MaintainAssignee: Justin Sherrill <jsherril>
Status: CLOSED ERRATA QA Contact: Gaurav Talreja <gtalreja>
Severity: high Docs Contact:
Priority: high    
Version: 6.9.0CC: ahumbe, apatel, desingh, jsherril, kgaikwad, myarboro, osousa, pcreech, saydas, zhunting
Target Milestone: 6.9.6Keywords: Triaged
Target Release: Unused   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: tfm-rubygem-katello-3.18.1.40-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-09-21 14:37:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Kenny Tordeurs 2021-08-16 09:19:03 UTC
Description of problem:
Upgrade fails stating the pulp3 content is already migrated.

--------------------------------------------------------------------------------
Switch support for certain content from Pulp 2 to Pulp 3: 
Performing final content migration before switching content           [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1:
 rake aborted!
The system appears to already be using pulp3 with all content migrated.
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.33/lib/katello/tasks/check_config.rake:15:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration => katello:check_config
(See full trace by running task with --trace)
Checking for valid Katello configuraton.
--------------------------------------------------------------------------------


Version-Release number of selected component (if applicable):
6.9.4
rubygem-foreman_maintain-0.8.10-1.el7sat.noarch

How reproducible:
100%

Steps to Reproduce:
1. All content is migrated before upgrade
2. satellite-maintain upgrade run --target-version 6.10
3. 

Actual results:
--------------------------------------------------------------------------------
Switch support for certain content from Pulp 2 to Pulp 3: 
Performing final content migration before switching content           [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1:
 rake aborted!
The system appears to already be using pulp3 with all content migrated.
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.33/lib/katello/tasks/check_config.rake:15:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration => katello:check_config
(See full trace by running task with --trace)
Checking for valid Katello configuraton.
--------------------------------------------------------------------------------


Expected results:
Upgrade to continue and be OK for the task as the content is already migrated

Additional info:

# foreman-rake katello:pulp3_migration --trace
** Invoke katello:pulp3_migration (first_time)
** Invoke environment (first_time)
** Execute environment
** Invoke katello:disable_dynflow (first_time)
** Execute katello:disable_dynflow
** Invoke katello:check_config (first_time)
** Invoke environment 
** Execute katello:check_config
Checking for valid Katello configuraton.
rake aborted!
The system appears to already be using pulp3 with all content migrated.
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.33/lib/katello/tasks/check_config.rake:15:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/task.rb:273:in `block in execute'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/task.rb:273:in `each'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/task.rb:273:in `execute'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/task.rb:214:in `block in invoke_with_call_chain'
/opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:235:in `mon_synchronize'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/task.rb:194:in `invoke_with_call_chain'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/task.rb:238:in `block in invoke_prerequisites'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/task.rb:236:in `each'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/task.rb:236:in `invoke_prerequisites'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/task.rb:213:in `block in invoke_with_call_chain'
/opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:235:in `mon_synchronize'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/task.rb:194:in `invoke_with_call_chain'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/task.rb:183:in `invoke'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/application.rb:160:in `invoke_task'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/application.rb:116:in `each'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/application.rb:116:in `block in top_level'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/application.rb:125:in `run_with_threads'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/application.rb:110:in `top_level'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/application.rb:83:in `block in run'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/application.rb:80:in `run'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
/opt/rh/rh-ruby25/root/usr/bin/rake:23:in `load'
/opt/rh/rh-ruby25/root/usr/bin/rake:23:in `<main>'
Tasks: TOP => katello:pulp3_migration => katello:check_config

# foreman-rake katello:check_config --trace
** Invoke katello:check_config (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute katello:check_config
Checking for valid Katello configuraton.
rake aborted!
The system appears to already be using pulp3 with all content migrated.
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.33/lib/katello/tasks/check_config.rake:15:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/task.rb:273:in `block in execute'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/task.rb:273:in `each'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/task.rb:273:in `execute'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/task.rb:214:in `block in invoke_with_call_chain'
/opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:235:in `mon_synchronize'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/task.rb:194:in `invoke_with_call_chain'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/task.rb:183:in `invoke'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/application.rb:160:in `invoke_task'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/application.rb:116:in `each'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/application.rb:116:in `block in top_level'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/application.rb:125:in `run_with_threads'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/application.rb:110:in `top_level'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/application.rb:83:in `block in run'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/lib/rake/application.rb:80:in `run'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
/opt/rh/rh-ruby25/root/usr/bin/rake:23:in `load'
/opt/rh/rh-ruby25/root/usr/bin/rake:23:in `<main>'
Tasks: TOP => katello:check_config

Comment 2 Sayan Das 2021-08-16 13:47:50 UTC
Yeah, having exact same situation here.

I performed pulp2 to 3 migration manually and hit this later during upgrade.

--------------------------------------------------------------------------------
Switch support for certain content from Pulp 2 to Pulp 3: 
Performing final content migration before switching content           [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1:
 rake aborted!
The system appears to already be using pulp3 with all content migrated.
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.32/lib/katello/tasks/check_config.rake:15:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration => katello:check_config
(See full trace by running task with --trace)
Checking for valid Katello configuraton.
--------------------------------------------------------------------------------
Scenario [Migration scripts to Satellite 6.10] failed.

The following steps ended up in failing state:

  [content-switchover]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-switchover"





So had to whitelist content-switchover to get past this but then upgrade was successful.

Comment 3 Sayan Das 2021-08-16 13:49:19 UTC
(In reply to Sayan Das from comment #2)
> Yeah, having exact same situation here.
> 
> I performed pulp2 to 3 migration manually and hit this later during upgrade.
> 
> -----------------------------------------------------------------------------
> ---
> Switch support for certain content from Pulp 2 to Pulp 3: 
> Performing final content migration before switching content           [FAIL]
> Failed executing foreman-rake katello:pulp3_migration, exit status 1:
>  rake aborted!
> The system appears to already be using pulp3 with all content migrated.
> /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.32/lib/katello/
> tasks/check_config.rake:15:in `block (2 levels) in <top (required)>'
> /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/exe/rake:27:in `<top
> (required)>'
> Tasks: TOP => katello:pulp3_migration => katello:check_config
> (See full trace by running task with --trace)
> Checking for valid Katello configuraton.
> -----------------------------------------------------------------------------
> ---
> Scenario [Migration scripts to Satellite 6.10] failed.
> 
> The following steps ended up in failing state:
> 
>   [content-switchover]
> 
> Resolve the failed steps and rerun
> the command. In case the failures are false positives,
> use --whitelist="content-switchover"
> 
> 
> 
> 
> 
> So had to whitelist content-switchover to get past this but then upgrade was
> successful.


If foreman-maintain can detect "The system appears to already be using pulp3 with all content migrated" , then it should not mark the step as failed. It should either handle it correctly and mark it as PASS or mark it as WARNING with some proper reasoning.

Comment 4 Justin Sherrill 2021-08-16 16:33:17 UTC
Connecting redmine issue https://projects.theforeman.org/issues/32831 from this bug

Comment 8 Justin Sherrill 2021-08-19 18:35:34 UTC
*** Bug 1995694 has been marked as a duplicate of this bug. ***

Comment 13 Gaurav Talreja 2021-09-09 15:15:17 UTC
Verified.

Tested on Satellite 6.9.6 Snap 1.0
Version: rubygem-foreman_maintain-0.8.13-2.el7sat.noarch

Steps:
1. Perform pulp2to3 migration
2. Run twice # foreman-maintain upgrade run --target-version 6.10 --whitelist="repositories-validate" (Fails post-switchover for repositories-setup)

Observation:
For upgrade reruns(which failed post-switchover), migration/switchover foreman-rake tasks just get skipped during the content-switchover step, like
 
--------------------------------------------------------------------------------
Switch support for certain content from Pulp 2 to Pulp 3:
Performing final content migration before switching content
Checking for valid Katello configuraton.
Switchover is already complete, skipping migration work.
Performing a check to verify everything that is needed has been migrated

Switching specified content over to pulp 3
Checking for valid Katello configuraton.
Switchover is already complete, skipping switchover task.
Re-running the installer to switch specified content over to pulp3    [OK]
--------------------------------------------------------------------------------

Comment 16 errata-xmlrpc 2021-09-21 14:37:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Satellite Maintenance 6.9.6 Async Bug Fix Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:3627

Comment 17 Red Hat Bugzilla 2023-09-15 01:13:43 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days