Bug 2219589
| Summary: | Upgrading to Satellite\Capsule 6.13.1 fails on "pulpcore-manager migrate --noinput" step due to Deadlock issues | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Satellite | Reporter: | Sayan Das <saydas> | ||||
| Component: | Pulp | Assignee: | satellite6-bugs <satellite6-bugs> | ||||
| Status: | CLOSED DUPLICATE | QA Contact: | Satellite QE Team <sat-qe-bz-list> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 6.13.2 | CC: | ahumbe, ashipati, jentrena, pbadguja, pdwyer, vijsingh | ||||
| Target Milestone: | Unspecified | Keywords: | Upgrades | ||||
| Target Release: | Unused | ||||||
| Hardware: | All | ||||||
| OS: | All | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2023-07-05 14:49:14 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Created attachment 1974001 [details]
txt file having the relevant logs when deadlock stuff happened.
This is perhaps a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=2203353 . I found it when I was searching for similar tracebacks in SFDC cases. We also have https://access.redhat.com/solutions/7011863 created on a fourth support case, JFYI. *** This bug has been marked as a duplicate of bug 2203353 *** |
Description of problem: Under certain circumstances, Upgrade to 6.13.1 can fail for Satellite as well as any external capsules, on the "pulpcore-manager migrate --noinput" step and it happens due to certain Deadlocks Version-Release number of selected component (if applicable): Satellite\Capsule 6.13.1 How reproducible: For specific customers only Steps to Reproduce ( Probable ): 1. Install a Satellite+Capsule 6.12 with lots of pulp and katello content. 2. Try upgrading the setup to 6.13. Actual results: 2023-07-03 11:36:04 [NOTICE] [configure] 2250 configuration steps out of 3004 steps complete. 2023-07-03 11:36:05 [NOTICE] [configure] 2500 configuration steps out of 3004 steps complete. 2023-07-03 11:36:05 [NOTICE] [configure] 2750 configuration steps out of 3004 steps complete. 2023-07-03 11:36:27 [ERROR ] [configure] 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0] 2023-07-03 11:36:27 [ERROR ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: change from 'notrun' to ['0'] failed: 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0] 2023-07-03 11:36:28 [NOTICE] [configure] 3000 configuration steps out of 3004 steps complete. 2023-07-03 11:36:31 [NOTICE] [configure] System configuration has finished. There were errors detected during install. Please address the errors and re-run the installer to ensure the system is properly configured. Failing to do so is likely to result in broken functionality. The full log is at /var/log/foreman-installer/satellite.log Package versions are being locked. [FAIL] Failed executing satellite-installer, exit status 6 -------------------------------------------------------------------------------- Scenario [Migration scripts to Satellite 6.13] failed. Expected results: No such errors: Additional info: From upgrade log: 2023-07-03 10:59:18 [DEBUG ] [configure] Executing with uid=pulp: 'pulpcore-manager migrate --noinput' 2023-07-03 10:59:39 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: Operations to perform: 2023-07-03 10:59:39 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: Apply all migrations: ansible, auth, certguard, container, contenttypes, core, file, rpm, sessions 2023-07-03 10:59:39 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: Running migrations: 2023-07-03 10:59:39 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: Applying core.0090_char_to_text_field...Traceback (most recent call last): 2023-07-03 10:59:39 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: File "/usr/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute 2023-07-03 10:59:39 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: return self.cursor.execute(sql, params) 2023-07-03 10:59:39 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: psycopg2.errors.DeadlockDetected: deadlock detected 2023-07-03 10:59:39 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: DETAIL: Process 119422 waits for AccessExclusiveLock on relation 1620850 of database 585335; blocked by process 2915. 2023-07-03 10:59:39 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: Process 2915 waits for AccessShareLock on relation 585593 of database 585335; blocked by process 119422. 2023-07-03 10:59:39 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: HINT: See server log for query details. 2023-07-03 10:59:39 [INFO ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: