Created attachment 1602796 [details]
strace of podman run
Description of problem:
After some destructive testing involving many reboots of a controller node, some of which via hard reset, podman got into a completely borked state. Namely every podman run command claims that there is no image:
# podman run -d --net=host --name=test 192.168.24.1:8787/rhosp15/openstack-haproxy:pcmklatest
must provide image ID and image name to use an image: invalid argument
This does not happen normally and it took a few reboots to get into this state, but in this state not a single run command works:
[root@controller-0 ~]# rpm -q podman kernel runc
Also note that we reproduced this also with a testing version of runc:
[root@controller-0 ~]# podman image inspect 41bfdd5a7361
error parsing image data "41bfdd5a7361b1ecd6233d67bd163008cb407f9098c99fb5e625f9918b1558ef": readlink /var/lib/containers/storage/overlay/l/7G7QCIMC7D5MK7NQXQC4WXJTV7: no such file or directory
Notice the uppercase there which seems a bit suspicious?
This seems very similar to https://github.com/code-ready/crc/issues/325 ?
Am attaching strace from the run command, the bolt db and ls -lR from /var/lib/containers
What sprints to the eye is that on a working node we have:
[root@controller-1 ~]# ls -l /var/lib/containers/storage/overlay/l/ |wc -l
Whereas on a broken node we have:
[root@controller-0 ~]# ls /var/lib/containers/storage/overlay/l/
Created attachment 1602797 [details]
ls -lr /var/lib/containers
Created attachment 1602798 [details]
bolt db on broken node
Thanks Michele for your feedback. Checked the code in vendor/github.com/containers/storage/drivers/overlay/overlay.go from podman-1.4.2-5.module+el8.1.0+4240+893c1ab8.src.rpm. The patches are already included. So set this to verified.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.