Bug 2030735

Summary: Satellite upgrade to 6.10 fails with error ' null value in column "repository_href" violates not-null constraint'
Product: Red Hat Satellite Reporter: rakesh kumar <rakumar>
Component: RepositoriesAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED WONTFIX QA Contact: Satellite QE Team <sat-qe-bz-list>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.10.1CC: ahumbe, alsouza, gpadholi, hyu, iballou, jsherril
Target Milestone: UnspecifiedKeywords: Triaged, Upgrades
Target Release: Unused   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2024-04-19 15:46:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description rakesh kumar 2021-12-09 15:34:35 UTC
Description of problem:

Satellite upgrade to 6.10 fails on pulp migration steps with below error:
~~~~~~~~~~~~~
~~~~~~~~~~~
ForemanTasks::TaskError: Task bdd39eaa-3298-43ec-996b-0164e25827ba: ActiveRecord::NotNullViolation: PG::NotNullViolation: ERROR:  null value in column "repository_href" violates not-null constraint
DETAIL:  Failing row contains (266, null, 20, 104).
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.46/lib/katello/tasks/pulp3_migration.rake:43:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
--------------------------------------------------------------------------------
Scenario [Migration scripts to Satellite 6.10] failed.

The following steps ended up in failing state:

  [content-switchover]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-switchover"
~~~~~~~~~~~~~~~~~~~~~ 

Version-Release number of selected component (if applicable):

Satellite 6.10


Steps to Reproduce:
Not sure in which scenario this error is reproducible 

Actual results:

Upgrade is failing 

Expected results:

The upgrade should be complete.

Additional info:

Comment 4 Justin Sherrill 2021-12-16 18:30:25 UTC
looking through the sosreport, i don't think we got the full logs that we need.  Getting the full traceback for 'ActiveRecord::NotNullViolation: PG::NotNullViolation: ERROR:  null value in column "repository_href" violates not-null constraint' would be helpful.  Providing all of /var/log/foreman/*production* should help.

In addition if we can get a database backup/reproducer, that would help quite a bit too.

Comment 5 Justin Sherrill 2022-02-09 15:25:05 UTC
the newest sosreport shows: 





We need to get more info.  It would be helpful to grab this:

#  sudo -u postgres psql pulpcore -c 'select pulp2_repo_id from pulp_2to3_migration_pulp2repository where pulp3_repository_id is null;'

This will output something like:

 pulp2_repo_id 
---------------
my-pulp2-id



if it outputs at least one, pick one from the list and run:

  sudo -u postgres psql foreman -c "select katello_root_repositories.name, katello_repositories.id from  katello_root_repositories inner join katello_repositories on katello_repositories.root_id = katello_root_repositories.id where katello_repositories.pulp_id = 'my-pulp2-id';"

Comment 6 Justin Sherrill 2022-02-09 15:28:01 UTC
Sorry the traceback showed:

2022-02-09T11:16:26 [E|bac|] PG::NotNullViolation: ERROR:  null value in column "repository_href" violates not-null constraint
 | DETAIL:  Failing row contains (301, null, 5, 189).
 |  (ActiveRecord::NotNullViolation)
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/postgresql_adapter.rb:675:in `async_exec_params'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/postgresql_adapter.rb:675:in `block (2 levels) in exec_no_cache'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'

SNIP
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/transactions.rb:212:in `transaction'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/transactions.rb:366:in `with_transaction_returning_status'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/persistence.rb:633:in `update!'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.46/app/services/katello/pulp3/migration.rb:240:in `import_repo'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.46/app/services/katello/pulp3/migration.rb:217:in `block in import_yum_repos'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/delegation.rb:87:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.46/app/services/katello/pulp3/migration.rb:200:in `each_with_index'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.46/app/services/katello/pulp3/migration.rb:200:in `import_yum_repos'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.46/app/services/katello/pulp3/migration.rb:187:in `import_repositories'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.46/app/services/katello/pulp3/migration.rb:109:in `block (3 levels) in import_pulp3_content'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.46/app/lib/katello/logging.rb:6:in `time'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.46/app/services/katello/pulp3/migration.rb:108:in `block (2 levels) in import_pulp3_content'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.46/app/services/katello/pulp3/migration.rb:107:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.46/app/services/katello/pulp3/migration.rb:107:in `block in import_pulp3_content'

Comment 19 Ian Ballou 2024-04-19 15:46:26 UTC
Closing this out since it relates to the old Pulp 2 to Pulp 3 migration. If you encounter this issue, please try `satellite-maintain content migration-reset` before retrying the upgrade. If that doesn't work, data will need to be cleaned on a per-user basis.