Bug 2231744 - "toolbox enter" fails with kernel 6.4.10 but it works properly on 6.4.9. Mitigation: create new toolbox containers with 6.4.10
Summary: "toolbox enter" fails with kernel 6.4.10 but it works properly on 6.4.9. Miti...
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: 38
Hardware: x86_64
OS: Linux
unspecified
low
Target Milestone: ---
Assignee: Kernel Maintainer List
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-08-13 20:06 UTC by Christopher Klooz
Modified: 2024-05-28 13:49 UTC (History)
17 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2024-05-28 13:49:07 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)

Description Christopher Klooz 2023-08-13 20:06:59 UTC
1. Please describe the problem:
When booting with kernel 6.4.10, "toolbox enter" fails with the following terminal output:
```
[username@fedora ~]$ toolbox enter
Error: failed to start container fedora-toolbox-38
[username@fedora ~]$ cat /proc/sys/kernel/tainted
0
[username@fedora ~]$ uname -r
6.4.10-200.fc38.x86_64
```

I expect that the following lines of the log are related to the error (see the log links below):
```
Aug 13 17:44:00 fedora podman[3464]: 2023-08-13 17:44:00.29005483 +0200 CEST m=+0.112119074 system refresh
Aug 13 17:44:00 fedora conmon[3552]: conmon 7756b6ffae401f0681c6 <error>: Failed to create container: exit status 1
Aug 13 17:44:02 fedora conmon[3631]: conmon 7756b6ffae401f0681c6 <error>: Failed to create container: exit status 1
```

The issue always appears when booting 6.4.10.
The issue never appears when booting 6.4.9.

But when I create a new container, the new one works, although it uses the same image: I configured the new container the same way as the old one -> create a "registry.fedoraproject.org/fedora-toolbox:38" container, install "rpmfusion-free-release" (I only installed VLC and its dependencies from this repo), and then do a dnf update (I keep updating my containers with dnf manually). Then I tested a movie with vlc: it worked fine.

Now, when booting 6.4.10, the old container still does not work (but the old one keeps working with 6.4.9), but the newly created container works fine also on 6.4.10:
```
[username@fedora ~]$ toolbox list
IMAGE ID      IMAGE NAME                                    CREATED
997b52ccbf85  registry.fedoraproject.org/fedora-toolbox:38  4 months ago

CONTAINER ID  CONTAINER NAME     CREATED      STATUS  IMAGE NAME
91245fa6482c  abc                2 hours ago  exited  registry.fedoraproject.org/fedora-toolbox:38
7756b6ffae40  fedora-toolbox-38  5 weeks ago  exited  registry.fedoraproject.org/fedora-toolbox:38
[username@fedora ~]$ toolbox enter 77
Error: failed to start container 77
[username@fedora ~]$ toolbox enter 91
⬢[username@toolbox ~]$ vlc
```

The issue remains after rebooting 6.4.10: The newly created container keeps working, while the old one keeps failing.

2. What is the Version-Release number of the kernel:
Affected kernel: 6.4.10 (x86_64). F38, KDE Spin, only default repositories, up to date as of today (all packages are from stable, except the kernel 6.4.10, which was still in testing when I installed it). "cat /proc/sys/kernel/tainted" = 0

3. Did it work previously in Fedora? If so, what kernel version did the issue
   *first* appear?  Old kernels are available for download at
   https://koji.fedoraproject.org/koji/packageinfo?packageID=8 :
The issue appears always with kernel-6.4.10-200.fc38.x86_64. When booting kernel-6.4.9-200.fc38.x86_64, everything works fine.

4. Can you reproduce this issue? If so, please provide the steps to reproduce
   the issue below:
See 1. I cannot determine if there is some specific property that has to be set in a "registry.fedoraproject.org/fedora-toolbox:38" container in order to cause the issue except that the container was created with a kernel / condition prior to 6.4.10. Maybe the cause lies in some podman/toolbox package and this package's state of the time when the affected container was created.

5. Does this problem occur with the latest Rawhide kernel? To install the
   Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by
   ``sudo dnf update --enablerepo=rawhide kernel``:
I cannot test this at the moment. If relevant, I can test it the week after next (let me know).

6. Are you running any modules that not shipped with directly Fedora's kernel?:
No. See above. Tainted=0. My Fedora does only use default repos natively (I use toolbox to keep it that way).

7. Please attach the kernel logs. You can get the complete kernel log
   for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the
   issue occurred on a previous boot, use the journalctl ``-b`` flag.
I have booted my machine once with 6.4.10, logged in, opened a terminal and then tried to "toolbox enter". After the failed output, I switched to a root terminal and stored the output of "journalctl --boot=0" and removed irrelevant contents in between booting and the failed try: https://gitlab.com/py0xc31/tmp71/-/raw/main/toolbox-kernel-error/extract_toolbox-fails_journalctl.--boot0.kernel-6.4.10.log

Then, I did the same with booting 6.4.9, and after the successful use of "toolbox enter", I again saved the journal in a root terminal (again with stuff in between removed): https://gitlab.com/py0xc31/tmp71/-/raw/main/toolbox-kernel-error/extract_toolbox-works_journalctl.--boot0.kernel-6.4.9.log

The two logs above have been created before I created the new container in 6.4.10. The next log was created after I created a new container in order to have both a failed and a successful case contained:

A full log (without anything removed) of "journalctl --no-hostname -k --boot=0" with 6.4.10 that I stored after a failed try with the affected container and a successful try with the newly created container can be found at: https://gitlab.com/py0xc31/tmp71/-/raw/main/toolbox-kernel-error/journalctl.--no-hostname.-k.--boot0.kernel-6.4.10


IMPORTANT:
I consider the issue solved as far as it concerns me because I could just replace the container. I write this report for your consideration, in case this problem indicates or is linked to a wider issue or so. I still retain the affected container in case you need me to test something or so (let me know).

If you consider this issue to have no wider impact, feel free to close the ticket.


Reproducible: Always

Comment 1 Aoife Moloney 2024-05-28 13:49:07 UTC
Fedora Linux 38 entered end-of-life (EOL) status on 2024-05-21.

Fedora Linux 38 is no longer maintained, which means that it
will not receive any further security or bug fix updates. As a result we
are closing this bug.

If you can reproduce this bug against a currently maintained version of Fedora Linux
please feel free to reopen this bug against that version. Note that the version
field may be hidden. Click the "Show advanced fields" button if you do not see
the version field.

If you are unable to reopen this bug, please file a new report against an
active release.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.