Bug 1960948
| Summary: | Error refreshing container XXX: error acquiring lock 0 for container | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Evgeni Golov <egolov> |
| Component: | podman | Assignee: | Jindrich Novy <jnovy> |
| Status: | CLOSED ERRATA | QA Contact: | Alex Jia <ajia> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | CentOS Stream | CC: | bbaude, bstinson, dwalsh, jligon, jnovy, jwboyer, lsm5, mheon, pthomas, tsweeney, umohnani, vrothber, ypu |
| Target Milestone: | beta | Keywords: | Reopened, Triaged |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | podman-3.3.0-0.11.el8 or newer | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-11-09 17:37:47 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1897579, 1989481 | ||
| Bug Blocks: | |||
|
Description
Evgeni Golov
2021-05-16 18:04:34 UTC
My guess would be this line: https://github.com/containers/podman/blob/a6a3df0273d19197286d12a805d7bc34c787b25f/vendor/github.com/containers/common/pkg/config/util_supported.go#L51 (from https://github.com/containers/common/blob/93838eeafc5408398eb1d93ca23c067f500190c9/pkg/config/util_supported.go#L51) Even tho I still don't understand why it doesn't use /run/user/1001 instead. Aha! $ podman info --log-level=DEBUG … DEBU[0000] Initializing boltdb state at /home/container/.local/share/containers/storage/libpod/bolt_state.db DEBU[0000] Overriding run root "/run/user/1001/containers" with "/tmp/podman-run-1001/containers" from database DEBU[0000] Overriding tmp dir "/run/user/1001/libpod/tmp" with "/tmp/run-1001/libpod/tmp" from database … runRoot: /tmp/podman-run-1001/containers So it seems something also needs to "migrate" the database? Unfortunately, the only way to migrate this is by complete removal of the DB (and all your containers and pods with it). Probably easier for us just to throw paths of that format in our tmpfiles.d upstream (you can fix locally by doing so now, until it gets picked up in a shipping package). (In reply to Matthew Heon from comment #3) > Unfortunately, the only way to migrate this is by complete removal of the DB > (and all your containers and pods with it). "migrate" :) But yeah, I can do that. Wonder what triggered it being set to those paths to begin with. > Probably easier for us just to throw paths of that format in our tmpfiles.d > upstream (you can fix locally by doing so now, until it gets picked up in a > shipping package). The original fix did change the paths in https://github.com/containers/podman/pull/8241 from run-* to podman-run-*, but seems not to have caught all the occurrences of "run-UID". And yeah, I did the local workaround for now here. Evgeni, given the discussion, this BZ looks like it can be closed as current release. Do you concur? Disagree. We need to update our tmpfiles.d to ignore the new pattern. No. At least with my limited knowledge, I see the following: * podman ships /usr/lib/tmpfiles.d/podman.conf that excludes /tmp/podman-run-* and /tmp/containers-user-* from being erased by systemd-tmpfiles * podman creates files in /tmp/podman-run-* *AND* in /tmp/run-* (the old path, pre https://github.com/containers/podman/pull/8241) * systemd-tmpfiles still removes files in /tmp/run-*, breaking podman We need to either * also exclude /tmp/run-* OR * make podman not ever use /tmp/run-* Podman 3.2 will no longer use /tmp/run, so this issue should be fixed with this release. Fixe in Podman v3.2 and RHEL 8.4.0.2 and higher. Assigning to Jindrich for any BZ/packaging needs. Negative. *New* Podman installs will not use `/tmp/run`. Existing ones will still used cached DB paths, which may include these paths. Needs to be moved back to Assigned Why, I think we can just tell older containers to be updated. This is rare, and will not happen going forward with new containers. (I believe). We can close it WONTFIX, or just state that these containers need to be updated. But I don't see us doing more engineering on it. Closing WONTFIX as per comment #13. (In reply to Daniel Walsh from comment #13) > Why, I think we can just tell older containers to be updated. This is rare, > and will not happen going forward with new containers. (I believe). > > We can close it WONTFIX, or just state that these containers need to be > updated. But I don't see us doing more engineering on it. Okay, but *how*? By wiping ~/.local/share/containers? At least that's how I read https://bugzilla.redhat.com/show_bug.cgi?id=1960948#c3 There is no way to update old containers. The only solution is to start from scratch with `podman system reset`. Because of this I am not in favor of closing. Based on the last comment, reopening and assigning back to Matt. Matt do you have a better idea of how to handle this? I need to check how versatile the Systemd tmpfiles syntax is - I want to say we can get away with adding `x /tmp/run-*/libpod` to our tmpfiles entry, but I'm not sure they do wildcards anywhere except the last character. Adding Valentin to the cc in case he's a thought or two. https://github.com/containers/podman/pull/10727 should fix, but I would appreciate further testing if anyone can. I verified that systemd-tmpfiles accepted the config but I'm not sure how to force it to reap directories, so I haven't verified it resolves this issue. This bug has been verified on podman-3.3.0-0.11.module+el8.5.0+11598+600219b6. [test@kvm-02-guest05 ~]$ podman run -d registry.access.redhat.com/rhel7 sleep infinity 4714d6d59160b1523fab3ea1bf4f898ab3b6dacfdab5c19049bb23e94a98ad3f [test@kvm-02-guest05 ~]$ rpm -q podman crun kernel podman-3.3.0-0.11.module+el8.5.0+11598+600219b6.x86_64 crun-0.20.1-1.module+el8.5.0+11621+c130ef1a.x86_64 kernel-4.18.0-316.el8.x86_64 [test@kvm-02-guest05 ~]$ grep libpod /usr/lib/tmpfiles.d/podman.conf x /tmp/run-*/libpod [test@kvm-02-guest05 ~]$ ls /tmp run-1000 systemd-private-14b25a9cb2d14ef7bcc24bfdbafc8df5-chronyd.service-ktw18f systemd-private-14b25a9cb2d14ef7bcc24bfdbafc8df5-postfix.service-wVWD8h [test@kvm-02-guest05 ~]$ podman run -d registry.access.redhat.com/rhel7 sleep infinity 4714d6d59160b1523fab3ea1bf4f898ab3b6dacfdab5c19049bb23e94a98ad3f [test@kvm-02-guest05 ~]$ podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 4714d6d59160 registry.access.redhat.com/rhel7:latest sleep infinity 6 seconds ago Up 7 seconds ago modest_engelbart [test@kvm-02-guest05 ~]$ ls /tmp run-1000 systemd-private-14b25a9cb2d14ef7bcc24bfdbafc8df5-chronyd.service-ktw18f systemd-private-14b25a9cb2d14ef7bcc24bfdbafc8df5-postfix.service-wVWD8h [test@kvm-02-guest05 ~]$ sudo /usr/bin/systemd-tmpfiles --clean [test@kvm-02-guest05 ~]$ ls /tmp run-1000 systemd-private-14b25a9cb2d14ef7bcc24bfdbafc8df5-chronyd.service-ktw18f systemd-private-14b25a9cb2d14ef7bcc24bfdbafc8df5-postfix.service-wVWD8h [test@kvm-02-guest05 ~]$ podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 4714d6d59160 registry.access.redhat.com/rhel7:latest sleep infinity 29 seconds ago Up 30 seconds ago modest_engelbart (In reply to Alex Jia from comment #24) [test@kvm-02-guest05 ~]$ grep 1m /usr/lib/tmpfiles.d/tmp.conf v /tmp 1777 root root 1m > [test@kvm-02-guest05 ~]$ sudo /usr/bin/systemd-tmpfiles --clean > [test@kvm-02-guest05 ~]$ ls /tmp > run-1000 This bug has been verified on podman-3.3.0-2.module+el8.5.0+12136+c1ac9593 w/ runc-1.0.1-4.module+el8.5.0+12048+8939a3ea. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: container-tools:rhel8 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:4154 |