Created attachment 1842541 [details] Tasks logs for reference Description of problem: Repository sync jobs are failing with the Exception "NoMethodError undefined method `repository_href' for nil:NilClass" post-upgrade to Satellite version 6.10 Version-Release number of selected component (if applicable): Satellite version 6.10 How reproducible: 100% Steps to Reproduce: 1. Upgraded satellite from 6.9 to 6.10 2. Enabled satellite tools 6.10 repository and was able to sync successfully. 3. Tried to sync old repositories, which we're enabled when the satellite was in 6.9. Actual results: Repository sync jobs for already enabled repositories failed with the below error:- Error:NoMethodError undefined method `repository_href' for nil:NilClass --- - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.35/app/services/katello/pulp3/repository.rb:236:in `sync'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.35/app/lib/actions/pulp3/repository/sync.rb:13:in `invoke_external_task'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action/polling.rb:84:in `initiate_external_action'" Expected results: Newly enabled repositories are syncing successfully. Expect the same for the already enabled repositories. Additional info:
For this bz, the user performing the upgrade completely whitelisted the migration and content switchover from pulp2 to pulp3 and so upgraded their satellite to pulp3 without doing the migration. There isn't really anything we can do after the fact except: * restore from backup * remove all CVVs, delete/disable all repositories I don't really see what we can do to fix this after the fact, and users shouldn't be whitelisting upgrade steps without knowing what they are doing. I propose we close as NOTABUG.
For this BZ, the plan is to stop suggesting users to whitelist failed steps,and instead suggest they contact support.
Thank you Justin for the review and update on this bugzilla. Whenever upgrade fails satellite maintain command suggests using --whitelist option to skip that step irrespective of whether is a critical step for successful upgrade or not. ================================== Resolve the failed steps and rerun the command. In case the failures are false positives, use --whitelist="content-switchover" ================================== ================================== Resolve the failed steps and rerun the command. In case the failures are false positives, use --whitelist="content-prepare" ================================== We would like to propose it to Foreman Maintain to improve the suggestion text in the error message something like: ================================== Resolve the failed steps and rerun the command. For any further help please contact Technical Support team. ================================== Amit, any thoughts on this?
Hello Ashish, Remove the steps from whitelist suggestion is good idea however I think we should not completely block the step from whitelisting as there could be still chances of having false errors? The foreman-maintain does not have the functionality to mark a step non whitelistable. The implementation can be in multiple ways like marking whole scenario as non whitelistable instead of a step. As a start I think we can start small and go for the step and the implementation is https://github.com/theforeman/foreman_maintain/pull/601 With above change I expect we are still able to use --whitelist="content-prepare" with upgrade command however we are not recommending it in the failure. Testing the pull request should help if its working as intended. Regards, Amit Upadhye.
Hi Amit, Thank you for creating PR where we can skip whitelist recommendations for Content Prepare and Content Switchover to start with. I will share it with the support team and see if they can help in reviewing or if they feel any more steps from the upgrade should have this option added. Regards, Ashish
For a reproducer: 1. install 6.9 2. do something that will cause the content migration to fail a. sync some content with the immediate download policy b. 'corrupt' some data echo "somedata" > /var/lib/pulp/content/path/to/package.rpm 3. try to upgrade to 6.10 You should see some text that says something like: The following steps ended up in failing state: [content-prepare] Resolve the failed steps and rerun the command. If the situation persists and, you are unclear what to do next, contact Red Hat Technical Support. In case the failures are false positives, use --whitelist="content-remove"
Moving to POST as upstream PR merged.
Steps: 1. Prepare Sat 6.11, sync some content and publish CV 2. Try content-switchover/content-prepare step # foreman-maintain content prepare 3. Try to fail any check, for example `repositories-validate` pre-upgrade check will fail till 6.11 GA # foreman-maintain upgrade check --target-version 6.11.z Observations: 1. content-switchover or content-prepare is supposed to fail here on 6.11, but a fix in foreman_maintain prevents the possibility of whitelisting this check, (check below example) and in the absence of the whitelist option user can't proceed for an upgrade with skipping a failed step. -------------------------------------------------------------------------------- Prepare content for Pulp 3: rake aborted! Don't know how to build task 'katello:pulp3_migration' (See the list of available tasks with `rake --tasks`) Did you mean? katello:import_subscriptions /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/exe/rake:27:in `<top (required)>' (See full trace by running task with --trace) [FAIL] Failed executing preserve_output=true foreman-rake katello:pulp3_migration, exit status 1 -------------------------------------------------------------------------------- Scenario [Prepare content for Pulp 3] failed. The following steps ended up in failing state: [content-prepare] 2. And for other checks in case of a fail state f-m asks the user the following message, -------------------------------------------------------------------------------- Validate availability of repositories: - Some repositories missing, calling `subscription-manager refresh` [FAIL] Following repositories are not available on your system: rhel-7-server-satellite-6.11-rpms, rhel-7-server-satellite-maintenance-6.11-rpms -------------------------------------------------------------------------------- Scenario [Checks before upgrading to Satellite 6.11.z] failed. The following steps ended up in failing state: [repositories-validate] Resolve the failed steps and rerun the command. If the situation persists and, you are unclear what to do next, contact Red Hat Technical Support. In case the failures are false positives, use --whitelist="repositories-validate"
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Satellite 6.11 Release), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:5498