Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2126697

Summary: containers config.json gets empty after sudden power loss
Product: Red Hat Enterprise Linux 8 Reporter: Suhaas Bhat <subhat>
Component: podmanAssignee: Jindrich Novy <jnovy>
Status: CLOSED ERRATA QA Contact: Alex Jia <ajia>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 8.6CC: arajendr, bbaude, dornelas, dwalsh, jligon, jnovy, lsm5, mheon, pthomas, tsweeney, umohnani, ypu
Target Milestone: rcKeywords: Triaged, ZStream
Target Release: ---Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: podman-4.2.1-3.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2130236 2130237 2136270 (view as bug list) Environment:
Last Closed: 2023-05-16 08:21:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2130236, 2130237, 2136270, 2136278    

Description Suhaas Bhat 2022-09-14 09:35:20 UTC
Description of problem:
After some tries of implementing #echo b > /proc/sysrq-trigger , /var/lib/containers/storage/overlay-containers/<container_id>/userdata/config.json
gets empty and the outcome is that we cannot see the inspect output which returns below error : 

Error: error unmarshalling container config: readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context

Version-Release number of selected component (if applicable):
podman-4.1.1-2.module+el8.6.0+15917+093ca6f8.x86_64

How reproducible:
Fully

Steps to Reproduce:
1. Install clean RHEL 8.6 with podman + netavark + runc
2. Spawn some containers using ubi8:minimal image (do not configure them to start on boot)
3. Cut the power / do `echo b > /proc/sysrq-trigger`
4. Run inspect on containers after regaining access
(if no errors found)
5. Start all containers
6. Repeat from step #3

I was able to reproduce this in second try.

Actual results:
Integrity of containers config.json is affected.

Expected results:
Containers data shouldn't be affected.

Additional info:

Comment 1 Matthew Heon 2022-09-14 13:36:33 UTC
There is no expectation that config.json is consistent and safe to use across reboots; it is rebuilt every time a container is started. The issue is that inspect is attempting to use the on-disk config.json even when it is not valid.

Comment 2 Matthew Heon 2022-09-14 13:44:16 UTC
Upstream fix via https://github.com/containers/podman/pull/15799

Comment 8 Matthew Heon 2022-09-23 15:40:16 UTC
Backport in progress via https://github.com/containers/podman/pull/15915

Comment 19 Alex Jia 2022-10-07 04:39:47 UTC
This bug has been verified on podman-4.2.1-3.module+el8.8.0+16760+2f749ab7 with netavark network backend.

[root@test-8-6-0-bz2126697 ~]# podman inspect foo | grep -iA2 args
WARN[0000] Error unmarshalling container 559f06b6c0779437ff516602e2f1330aa03ced98439a066ae2cb411b107d99b4 config: readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||... 
          "Args": [
               "infinity"
          ],

[root@test-8-6-0-bz2126697 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.6 (Ootpa)

[root@test-8-6-0-bz2126697 ~]# rpm -q podman runc netavark systemd kernel
podman-4.2.1-3.module+el8.8.0+16760+2f749ab7.x86_64
runc-1.1.4-1.module+el8.8.0+16760+2f749ab7.x86_64
netavark-1.0.1-35.module+el8.6.0+15917+093ca6f8.x86_64
systemd-239-58.el8.x86_64
kernel-4.18.0-372.9.1.el8.x86_64

[root@test-8-6-0-bz2126697 ~]# grep network_backend /usr/share/containers/containers.conf
#network_backend = ""
network_backend = "netavark"

Comment 20 Tom Sweeney 2022-10-19 19:20:31 UTC
*** Bug 2136270 has been marked as a duplicate of this bug. ***

Comment 22 errata-xmlrpc 2023-05-16 08:21:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: container-tools:rhel8 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:2758