Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
Under certain circumstances, Upgrade to 6.13.1 can fail for Satellite as well as any external capsules, on the "pulpcore-manager migrate --noinput" step and it happens due to certain Deadlocks
Version-Release number of selected component (if applicable):
Satellite\Capsule 6.13.1
How reproducible:
For specific customers only
Steps to Reproduce ( Probable ):
1. Install a Satellite+Capsule 6.12 with lots of pulp and katello content.
2. Try upgrading the setup to 6.13.
Actual results:
2023-07-03 11:36:04 [NOTICE] [configure] 2250 configuration steps out of 3004 steps complete.
2023-07-03 11:36:05 [NOTICE] [configure] 2500 configuration steps out of 3004 steps complete.
2023-07-03 11:36:05 [NOTICE] [configure] 2750 configuration steps out of 3004 steps complete.
2023-07-03 11:36:27 [ERROR ] [configure] 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
2023-07-03 11:36:27 [ERROR ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: change from 'notrun' to ['0'] failed: 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
2023-07-03 11:36:28 [NOTICE] [configure] 3000 configuration steps out of 3004 steps complete.
2023-07-03 11:36:31 [NOTICE] [configure] System configuration has finished.
There were errors detected during install.
Please address the errors and re-run the installer to ensure the system is properly configured.
Failing to do so is likely to result in broken functionality.
The full log is at /var/log/foreman-installer/satellite.log
Package versions are being locked.
[FAIL]
Failed executing satellite-installer, exit status 6
--------------------------------------------------------------------------------
Scenario [Migration scripts to Satellite 6.13] failed.
Expected results:
No such errors:
Additional info:
From upgrade log:
2023-07-03 10:59:18 [DEBUG ] [configure] Executing with uid=pulp: 'pulpcore-manager migrate --noinput'
2023-07-03 10:59:39 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: Operations to perform:
2023-07-03 10:59:39 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: Apply all migrations: ansible, auth, certguard, container, contenttypes, core, file, rpm, sessions
2023-07-03 10:59:39 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: Running migrations:
2023-07-03 10:59:39 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: Applying core.0090_char_to_text_field...Traceback (most recent call last):
2023-07-03 10:59:39 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: File "/usr/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
2023-07-03 10:59:39 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: return self.cursor.execute(sql, params)
2023-07-03 10:59:39 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: psycopg2.errors.DeadlockDetected: deadlock detected
2023-07-03 10:59:39 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: DETAIL: Process 119422 waits for AccessExclusiveLock on relation 1620850 of database 585335; blocked by process 2915.
2023-07-03 10:59:39 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: Process 2915 waits for AccessShareLock on relation 585593 of database 585335; blocked by process 119422.
2023-07-03 10:59:39 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: HINT: See server log for query details.
2023-07-03 10:59:39 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:
Description of problem: Under certain circumstances, Upgrade to 6.13.1 can fail for Satellite as well as any external capsules, on the "pulpcore-manager migrate --noinput" step and it happens due to certain Deadlocks Version-Release number of selected component (if applicable): Satellite\Capsule 6.13.1 How reproducible: For specific customers only Steps to Reproduce ( Probable ): 1. Install a Satellite+Capsule 6.12 with lots of pulp and katello content. 2. Try upgrading the setup to 6.13. Actual results: 2023-07-03 11:36:04 [NOTICE] [configure] 2250 configuration steps out of 3004 steps complete. 2023-07-03 11:36:05 [NOTICE] [configure] 2500 configuration steps out of 3004 steps complete. 2023-07-03 11:36:05 [NOTICE] [configure] 2750 configuration steps out of 3004 steps complete. 2023-07-03 11:36:27 [ERROR ] [configure] 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0] 2023-07-03 11:36:27 [ERROR ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: change from 'notrun' to ['0'] failed: 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0] 2023-07-03 11:36:28 [NOTICE] [configure] 3000 configuration steps out of 3004 steps complete. 2023-07-03 11:36:31 [NOTICE] [configure] System configuration has finished. There were errors detected during install. Please address the errors and re-run the installer to ensure the system is properly configured. Failing to do so is likely to result in broken functionality. The full log is at /var/log/foreman-installer/satellite.log Package versions are being locked. [FAIL] Failed executing satellite-installer, exit status 6 -------------------------------------------------------------------------------- Scenario [Migration scripts to Satellite 6.13] failed. Expected results: No such errors: Additional info: From upgrade log: 2023-07-03 10:59:18 [DEBUG ] [configure] Executing with uid=pulp: 'pulpcore-manager migrate --noinput' 2023-07-03 10:59:39 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: Operations to perform: 2023-07-03 10:59:39 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: Apply all migrations: ansible, auth, certguard, container, contenttypes, core, file, rpm, sessions 2023-07-03 10:59:39 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: Running migrations: 2023-07-03 10:59:39 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: Applying core.0090_char_to_text_field...Traceback (most recent call last): 2023-07-03 10:59:39 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: File "/usr/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute 2023-07-03 10:59:39 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: return self.cursor.execute(sql, params) 2023-07-03 10:59:39 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: psycopg2.errors.DeadlockDetected: deadlock detected 2023-07-03 10:59:39 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: DETAIL: Process 119422 waits for AccessExclusiveLock on relation 1620850 of database 585335; blocked by process 2915. 2023-07-03 10:59:39 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: Process 2915 waits for AccessShareLock on relation 585593 of database 585335; blocked by process 119422. 2023-07-03 10:59:39 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: HINT: See server log for query details. 2023-07-03 10:59:39 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: