Bug 2240185
| Summary: | Upgrade [OSP16.2 ->OSP17.1] undercloud FFU failed cleaning the mysql transient due to "unexpected end of JSON input" | |||
|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Juan Badia Payno <jbadiapa> | |
| Component: | openstack-tripleo-common | Assignee: | Nobody <nobody> | |
| Status: | CLOSED ERRATA | QA Contact: | Khomesh Thakre <kthakre> | |
| Severity: | urgent | Docs Contact: | ||
| Priority: | urgent | |||
| Version: | 16.2 (Train) | CC: | bshephar, fpantano, jelynch, jmarcian, jpretori, kthakre, lbezdick, mariel, mburns, mflusche, nkawamot, prgutier, ramishra, slinaber, tvignaud | |
| Target Milestone: | z6 | Keywords: | Reopened, Triaged | |
| Target Release: | 16.2 (Train on RHEL 8.4) | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | openstack-tripleo-common-11.7.1-2.20230809225404.e189622.el8ost | Doc Type: | No Doc Update | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 2241029 2241918 (view as bug list) | Environment: | ||
| Last Closed: | 2023-11-08 19:19:17 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 2241029, 2241918 | |||
The bz#2102078 might me related [root@undercloud-0 ~]# skopeo inspect --raw docker://undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-mariadb:16.2_20230509.1 | python3 -m json.tool | grep schema "schemaVersion": 1, [root@undercloud-0 ~]# skopeo inspect --raw docker://rhos-qe-mirror.lab.eng.rdu2.redhat.com:5002/rh-osbs/rhosp16-openstack-mariadb:16.2_20230509.1 | python3 -m json.tool | grep schema "schemaVersion": 2 [root@undercloud-0 ~]# podman images | grep maria undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-mariadb 16.2_20230509.1 865d60d6ee90 4 months ago 780 MB [root@undercloud-0 ~]# podman inspect 865d60d6ee90 Error: unexpected end of JSON input [root@undercloud-0 ~]# dnf module switch-to container-tools:3.0 [root@undercloud-0 ~]# podman inspect 865d60d6ee90 [ { "Id": "865d60d6ee9080b41ce8c1979428bdc4465337103c2d9c79a02151efead63e92", "Digest": "sha256:2b6a6acc625697f28dc2e87972e99a6167cd8c97e17430b3e9194ebfba451a25", . . } ] [1] https://bugzilla.redhat.com/show_bug.cgi?id=2102078 Yeah, I think this is a duplicate of that BZ. We already have two other bz's about the issue opened at the same time, I suspect they're all related to the Quay update. Let's track the issue on 2240163 and see if the solution posted there resolves the issue. *** This bug has been marked as a duplicate of bug 2240163 *** Reopening as the fix needs to be done into 16.2 and this is 16.2 error even if it's in middle of upgrade. *** Bug 2242366 has been marked as a duplicate of this bug. *** *** Bug 2243668 has been marked as a duplicate of this bug. *** Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 16.2.6 (Train) bug fix and enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2023:6307 |
Doing the undercloud FFU on a ganesha [1] the following error showed up during the "openstack undercloud upgrade" 2023-09-21 23:00:37 | 2023-09-21 23:00:37.069891 | 525400e7-0603-403f-baed-00000000095d | FATAL | Clean up redo log by running a transient mysql server | undercloud-0 | error={"changed": true, "cmd": "podman run --rm -u root --net=host -e \"KOLLA_CONFIG_STRATEGY=COPY_ALWAYS\" -v /var/lib/kolla/config_files/mysql.json:/var/lib/kolla/config_files/config.json -v /var/lib/config-data/puppet-generated/mysql:/var/lib/kolla/config_files/src:ro -v /etc/localtime:/etc/localtime:ro -v /etc/hosts:/etc/hosts:ro -v /var/lib/mysql:/var/lib/mysql -v /var/log/containers/mysql:/var/log/mariadb \"undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-mariadb:16.2_20230509.1\" /bin/bash -ecx \"kolla_set_configs; mysqld_safe --user=mysql --skip-networking --log-error=/var/log/mariadb/mariadb-upgrade.log & timeout 180 sh -c 'while ! mysqladmin ping --silent; do sleep 1; done'; mysqladmin shutdown\"", "delta": "0:00:00.136183", "end": "2023-09-21 23:00:37.042725", "msg": "non-zero return code", "rc": 125, "start": "2023-09-21 23:00:36.906542", "stderr": "Error: unexpected end of JSON input", "stderr_lines": ["Error: unexpected end of JSON input"], "stdout": "", "stdout_lines": []} 2023-09-21 23:00:37 | 2023-09-21 23:00:37.071476 | 525400e7-0603-403f-baed-00000000095d | TIMING | Clean up redo log by running a transient mysql server | undercloud-0 | 0:06:24.263487 | 0.39s The puddle used was RHOS-17.1-RHEL-8-20230907.n.1 The full process can be seen at [2] [1] [2] can be seen on the comment 2