RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2126697 - containers config.json gets empty after sudden power loss
Summary: containers config.json gets empty after sudden power loss
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: podman
Version: 8.6
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: rc
: ---
Assignee: Jindrich Novy
QA Contact: Alex Jia
URL:
Whiteboard:
: 2136270 (view as bug list)
Depends On:
Blocks: 2130236 2130237 2136270 2136278
TreeView+ depends on / blocked
 
Reported: 2022-09-14 09:35 UTC by Suhaas Bhat
Modified: 2023-05-16 09:09 UTC (History)
12 users (show)

Fixed In Version: podman-4.2.1-3.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 2130236 2130237 2136270 (view as bug list)
Environment:
Last Closed: 2023-05-16 08:21:12 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github containers podman pull 15799 0 None Merged Ensure that a broken OCI spec does not break inspect 2022-09-26 09:41:18 UTC
Red Hat Issue Tracker RHELPLAN-133909 0 None None None 2022-10-07 03:43:44 UTC
Red Hat Product Errata RHSA-2023:2758 0 None None None 2023-05-16 08:22:56 UTC

Description Suhaas Bhat 2022-09-14 09:35:20 UTC 
Description of problem:
After some tries of implementing #echo b > /proc/sysrq-trigger , /var/lib/containers/storage/overlay-containers/<container_id>/userdata/config.json
gets empty and the outcome is that we cannot see the inspect output which returns below error : 

Error: error unmarshalling container config: readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context

Version-Release number of selected component (if applicable):
podman-4.1.1-2.module+el8.6.0+15917+093ca6f8.x86_64

How reproducible:
Fully

Steps to Reproduce:
1. Install clean RHEL 8.6 with podman + netavark + runc
2. Spawn some containers using ubi8:minimal image (do not configure them to start on boot)
3. Cut the power / do `echo b > /proc/sysrq-trigger`
4. Run inspect on containers after regaining access
(if no errors found)
5. Start all containers
6. Repeat from step #3

I was able to reproduce this in second try.

Actual results:
Integrity of containers config.json is affected.

Expected results:
Containers data shouldn't be affected.

Additional info:
Comment 1 Matthew Heon 2022-09-14 13:36:33 UTC 
There is no expectation that config.json is consistent and safe to use across reboots; it is rebuilt every time a container is started. The issue is that inspect is attempting to use the on-disk config.json even when it is not valid.
Comment 2 Matthew Heon 2022-09-14 13:44:16 UTC 
Upstream fix via https://github.com/containers/podman/pull/15799
Comment 8 Matthew Heon 2022-09-23 15:40:16 UTC 
Backport in progress via https://github.com/containers/podman/pull/15915
Comment 19 Alex Jia 2022-10-07 04:39:47 UTC 
This bug has been verified on podman-4.2.1-3.module+el8.8.0+16760+2f749ab7 with netavark network backend.

[root@test-8-6-0-bz2126697 ~]# podman inspect foo | grep -iA2 args
WARN[0000] Error unmarshalling container 559f06b6c0779437ff516602e2f1330aa03ced98439a066ae2cb411b107d99b4 config: readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||... 
          "Args": [
               "infinity"
          ],

[root@test-8-6-0-bz2126697 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.6 (Ootpa)

[root@test-8-6-0-bz2126697 ~]# rpm -q podman runc netavark systemd kernel
podman-4.2.1-3.module+el8.8.0+16760+2f749ab7.x86_64
runc-1.1.4-1.module+el8.8.0+16760+2f749ab7.x86_64
netavark-1.0.1-35.module+el8.6.0+15917+093ca6f8.x86_64
systemd-239-58.el8.x86_64
kernel-4.18.0-372.9.1.el8.x86_64

[root@test-8-6-0-bz2126697 ~]# grep network_backend /usr/share/containers/containers.conf
#network_backend = ""
network_backend = "netavark"
Comment 20 Tom Sweeney 2022-10-19 19:20:31 UTC 
*** Bug 2136270 has been marked as a duplicate of this bug. ***
Comment 22 errata-xmlrpc 2023-05-16 08:21:12 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: container-tools:rhel8 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:2758
Note You need to log in before you can comment on or make changes to this bug.