Bug 2225011

Summary: [OSP 16.2 -> 17.1] Undercloud upgrade fails on 'migrate existing introspection data' with lost connection to mysql.
Product: Red Hat OpenStack Reporter: Ravi Singh <ravsingh>
Component: documentationAssignee: kgilliga
Status: CLOSED CURRENTRELEASE QA Contact: RHOS Documentation Team <rhos-docs>
Severity: medium Docs Contact:
Priority: medium    
Version: 17.1 (Wallaby)CC: jbadiapa, jpretori, lbezdick, mburns, sgolovat
Target Milestone: ---Keywords: Documentation, Reopened, Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2024-07-30 13:29:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ravi Singh 2023-07-24 08:59:38 UTC
Description of problem:

Re-run of undercloud upgrade(16.2->17.1) failed with following error:

+++
2023-07-19 08:15:18.869778 | 525400cf-381f-5864-38c7-00000000073e |      FATAL | migrate existing introspection data | undercloud | error={"changed": true, "cmd": "podman exec -u root ironic_inspector ironic-inspector-migrate-data --from swift --to database --config-file /etc/ironic-inspector/inspector.conf\n", "delta": "0:01:42.467282", "end": "2023-07-19 08:15:18.830981", "failed_when_result": true, "msg": "non-zero return code", "rc": 1, "start": "2023-07-19 08:13:36.363699", "stderr": "b\"(pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')\\n(Background on this error at: http://sqlalche.me/e/e3q8)\"", "stderr_lines":["b\"(pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')\\n(Background on this error at: http://sqlalche.me/e/e3q8)\""], "stdout": "", "stdout_lines": []}
+++

Version-Release number of selected component (if applicable):
17.1

How reproducible:
Everytime

Steps to Reproduce:

1. Run 1st time undercloud upgrade which failed with some reason(in this case it failed while pulling images due to wrong tagging).
2. Re-run undercloud upgrade  after fixing issue in step-1. Upgrade failed on

~~~
2023-07-19 08:15:18.869778 | 525400cf-381f-5864-38c7-00000000073e |      FATAL | migrate existing introspection data | undercloud | error={"changed": true, "cmd": "podman exec -u root ironic_inspector ironic-inspector-migrate-data --from swift --to database --config-file /etc/ironic-inspector/inspector.conf\n", "delta": "0:01:42.467282", "end": "2023-07-19 08:15:18.830981", "failed_when_result": true, "msg": "non-zero return code", "rc": 1, "start": "2023-07-19 08:13:36.363699", "stderr": "b\"(pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')\\n(Background on this error at: http://sqlalche.me/e/e3q8)\"", "stderr_lines":["b\"(pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')\\n(Background on this error at: http://sqlalche.me/e/e3q8)\""], "stdout": "", "stdout_lines": []}
~~~
3. Retart tripleo_mysql.service & upgrade passes.
Actual results:

Upgrade failed during re-run.

Expected results:

Should pass.

Additional info: