Bug 2240163

I compared recent passing and failing CI builds and I see several differenced in minor package number (though the OSP build version is the same, 2 weeks difference). Especially since this seems to be podman related, the version got bumped 2:4.4.1-12.el9_2 -> 2:4.4.1-13.el9_2 . That one got change https://github.com/containers/podman/commit/dd490ff plus seems to be related to https://bugzilla.redhat.com/show_bug.cgi?id=2229749 - might be worth looking at as part of the debugging.

*** Bug 2240252 has been marked as a duplicate of this bug. ***

*** Bug 2240185 has been marked as a duplicate of this bug. ***

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 17.0 & 17.1 Bug Fix Advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:5364

Hello, 

This issue still present with openstack-tripleo-common-15.4.1-1.20230518205258.el8ost.noarch [1] and is currently preventing the upgrade of an Undercloud from 16.2.5 -> 17.1 during this task on the Undercloud upgrade[2],[3]. 




Should this bz be reopened or should I create a new BZ ? 




[1] 
---
[stack@osp-dir ~]$ rpm -qi openstack-tripleo-common-15.4.1-1.20230518205258.el8ost.noarch
Name        : openstack-tripleo-common
Version     : 15.4.1
Release     : 1.20230518205258.el8ost
Architecture: noarch
Install Date: Wed 18 Oct 2023 09:54:25 AM EDT
Group       : Unspecified
Size        : 75426
License     : ASL 2.0
Signature   : RSA/SHA256, Mon 25 Sep 2023 03:00:26 PM EDT, Key ID 199e2f91fd431d51
Source RPM  : openstack-tripleo-common-15.4.1-1.20230518205258.el8ost.src.rpm
Build Date  : Mon 25 Sep 2023 02:42:51 PM EDT
Build Host  : s390-066.build.eng.bos.redhat.com
Relocations : (not relocatable)
Packager    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
Vendor      : Red Hat, Inc.
URL         : https://github.com/rdo-management/tripleo-common
Summary     : Python library for code used by TripleO projects.
Description :
Python library for code used by TripleO projects.



[2] 
---
2023-10-18 10:06:07.236779 | 52540015-7743-1eb7-458d-000000000952 |       TASK | Clean up redo log by running a transient mysql server
2023-10-18 10:06:07.675856 | 52540015-7743-1eb7-458d-000000000952 |      FATAL | Clean up redo log by running a transient mysql server | osp-dir | error={"changed": true, "cmd": "podman run --rm -u root --net=host -e \"KOLLA_CONFIG_STRATEGY=COPY_ALWAYS\" -v /var/lib/kolla/config_files/mysql.json:/var/lib/kolla/config_files/config.json -v /var/lib/config-data/puppet-generated/mysql:/var/lib/kolla/config_files/src:ro -v /etc/localtime:/etc/localtime:ro -v /etc/hosts:/etc/hosts:ro -v /var/lib/mysql:/var/lib/mysql -v /var/log/containers/mysql:/var/log/mariadb \"osp-dir.ctlplane.voltron.xyz:8787/rhosp-rhel8/openstack-mariadb:16.2.5\" /bin/bash -ecx \"kolla_set_configs; mysqld_safe --user=mysql --skip-networking --log-error=/var/log/mariadb/mariadb-upgrade.log & timeout 180 sh -c 'while ! mysqladmin ping --silent; do sleep 1; done'; mysqladmin shutdown\"", "delta": "0:00:00.146240", "end": "2023-10-18 10:06:07.640817", "msg": "non-zero return code", "rc": 125, "start": "2023-10-18 10:06:07.494577", "stderr": "Error: unexpected end of JSON input", "stderr_lines": ["Error: unexpected end of JSON input"], "stdout": "", "stdout_lines": []}
2023-10-18 10:06:07.677828 | 52540015-7743-1eb7-458d-000000000952 |     TIMING | Clean up redo log by running a transient mysql server | osp-dir | 0:05:34.157672 | 0.44s

PLAY RECAP *********************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0
osp-dir                    : ok=245  changed=93   unreachable=0    failed=1    skipped=133  rescued=0    ignored=2



[3]
---
[stack@osp-dir ~]$ sudo podman --log-level=debug image inspect osp-dir.ctlplane.voltron.xyz:8787/rhosp-rhel8/openstack-mariadb:16.2.5
INFO[0000] podman filtering at log level debug
DEBU[0000] Called inspect.PersistentPreRunE(podman --log-level=debug image inspect osp-dir.ctlplane.voltron.xyz:8787/rhosp-rhel8/openstack-mariadb:16.2.5)
DEBU[0000] Merged system config "/usr/share/containers/containers.conf"
DEBU[0000] Merged system config "/etc/containers/containers.conf"
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /var/lib/containers/storage
DEBU[0000] Using run root /run/containers/storage
DEBU[0000] Using static dir /var/lib/containers/storage/libpod
DEBU[0000] Using tmp dir /run/libpod
DEBU[0000] Using volume path /var/lib/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] cached value indicated that overlay is supported
DEBU[0000] cached value indicated that metacopy is being used
DEBU[0000] cached value indicated that native-diff is not being used
INFO[0000] Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true
DEBU[0000] Initializing event backend journald
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/runc"
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist
DEBU[0000] Default CNI network name podman is unchangeable
INFO[0000] Setting parallel job count to 25
DEBU[0000] Looking up image "osp-dir.ctlplane.voltron.xyz:8787/rhosp-rhel8/openstack-mariadb:16.2.5" in local containers storage
DEBU[0000] Trying "osp-dir.ctlplane.voltron.xyz:8787/rhosp-rhel8/openstack-mariadb:16.2.5" ...
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@414957cc322bd69c9e88b9ae66cd01ba07ea13826d47c5b756bd1b8847cba7e2"
DEBU[0000] Found image "osp-dir.ctlplane.voltron.xyz:8787/rhosp-rhel8/openstack-mariadb:16.2.5" as "osp-dir.ctlplane.voltron.xyz:8787/rhosp-rhel8/openstack-mariadb:16.2.5" in local containers storage
DEBU[0000] Inspecting image 414957cc322bd69c9e88b9ae66cd01ba07ea13826d47c5b756bd1b8847cba7e2
Error: unexpected end of JSON input

(In reply to Alberto Rivera Laporte from comment #25)
> Hello, 
> 
> This issue still present with
> openstack-tripleo-common-15.4.1-1.20230518205258.el8ost.noarch [1] and is
> currently preventing the upgrade of an Undercloud from 16.2.5 -> 17.1 during
> this task on the Undercloud upgrade[2],[3]. 
> 
> 
> 
> 
> Should this bz be reopened or should I create a new BZ ? 

Please open a new issue as this one has already been associated with an errata as part of shipped product, and it will make it difficult to address.

It'd be ideal to have a separate issue with the appropriate information identifying where the problem lies along with a new description in order to avoid having to trawl the entire backlog for cleanliness.

> Should this bz be reopened or should I create a new BZ ? 

There is no need to create a new BZ or re-open this one. This BZ does not address related (16.2->17.1)upgrade issues  and that instead has been addressed with https://bugzilla.redhat.com/show_bug.cgi?id=2240185. The issue is during upgrade we try to inspect old 16.2 local images with new podman.

*** Bug 2240252 has been marked as a duplicate of this bug. ***