Hide Forgot
Description of problem: Errors build layer on registry.redhat.io/rhel8-beta/rhel-minimal with host subscribed using subscription-manager, this is the image we just released publicly for RHEL-8 Beta Version-Release number of selected component (if applicable): Host installed with our released RHEL-8-Beta microdnf-3.0.1-1.el8.x86_64 How reproducible: 100% - exact same issue seen on two architectures. Steps to Reproduce: - Install RHEL-8-Beta host (Version released in production) - install container tools module - login to terms based registry (using Customer Portal login) podman login registry.redhat.io - subscribe host using subscription-manager - create the following Dockerfile # mkdir traceroute; cd traceroute # cat << EOF Dockerfile # podman build --rm --no-cache --force-rm -t traceroute . # podman run --rm -it traceroute # podman rmi traceroute FROM registry.redhat.io/rhel8-beta/rhel-minimal MAINTAINER Martin Jenner "mjenner" RUN microdnf install --enablerepo=rhel-8-for-x86_64-baseos-beta-rpms traceroute CMD ["/bin/traceroute","google.com"] EOF - use the command in the Dockerfile above to build a layer on the rhel-minimal image Actual results: Errors when building layer on the image # podman build --rm --no-cache --force-rm -t traceroute . STEP 1: FROM registry.redhat.io/rhel8-beta/rhel-minimal STEP 2: MAINTAINER Martin Jenner "mjenner" ERRO[0000] HOSTNAME is not supported for OCI image format, hostname 52abe2f52726 will be ignored. Must use `docker` format --> e7a1730d924dc0dd11833be898bff36dff2a93d97c8bca409eb07400f17d9884 STEP 3: FROM e7a1730d924dc0dd11833be898bff36dff2a93d97c8bca409eb07400f17d9884 STEP 4: RUN microdnf install --enablerepo=rhel-8-for-x86_64-baseos-beta-rpms traceroute Downloading metadata... Downloading metadata... Package Repository Size Installing: traceroute-3:2.1.0-6.el8.x86_64 rhel-8-for-x86_ 69.1 kB Transaction Summary: Installing: 1 packages Reinstalling: 0 packages Upgrading: 0 packages Removing: 0 packages Downgrading: 0 packages Downloading packages... Running transaction test... Installing: traceroute;3:2.1.0-6.el8;x86_64;rhel-8-for-x86_64-baseos-beta-rpms Complete. ERRO[0027] Can't add file /var/lib/containers/storage/overlay/a7cd6bc213e4721bb62431ca654358710222a1e023f99943decfaa9d9ddd6f00/diff/var/cache/yum/metadata/rhel-8-for-x86_64-appstream-beta-rpms-8-x86_64/gpgdir/S.gpg-agent to tar: archive/tar: sockets not supported ERRO[0027] Can't add file /var/lib/containers/storage/overlay/a7cd6bc213e4721bb62431ca654358710222a1e023f99943decfaa9d9ddd6f00/diff/var/cache/yum/metadata/rhel-8-for-x86_64-appstream-beta-rpms-8-x86_64/gpgdir/S.gpg-agent.browser to tar: archive/tar: sockets not supported ERRO[0027] Can't add file /var/lib/containers/storage/overlay/a7cd6bc213e4721bb62431ca654358710222a1e023f99943decfaa9d9ddd6f00/diff/var/cache/yum/metadata/rhel-8-for-x86_64-appstream-beta-rpms-8-x86_64/gpgdir/S.gpg-agent.extra to tar: archive/tar: sockets not supported ERRO[0027] Can't add file /var/lib/containers/storage/overlay/a7cd6bc213e4721bb62431ca654358710222a1e023 Expected results: Layer should build successfully on top of image. Additional info: If I run the exact same test without the host subscribed and use a private repo to install traceroute I do not see the errors.
Slightly corrected instructions: # mkdir traceroute; cd traceroute # cat << EOF > Dockerfile FROM registry.redhat.io/rhel8-beta/rhel-minimal MAINTAINER Martin Jenner "mjenner" RUN microdnf install --enablerepo=rhel-8-for-x86_64-baseos-beta-rpms traceroute CMD ["/bin/traceroute","google.com"] EOF # podman build --rm --no-cache --force-rm -t traceroute . I was able to reproduce with same output: STEP 1: FROM registry.redhat.io/rhel8-beta/rhel-minimal STEP 2: MAINTAINER Martin Jenner "mjenner" ERRO[0000] HOSTNAME is not supported for OCI image format, hostname 52abe2f52726 will be ignored. Must use `docker` format --> f202b496dac42e29e5979945b2818a588d5ef6593194f342fc40703f7f95cbed STEP 3: FROM f202b496dac42e29e5979945b2818a588d5ef6593194f342fc40703f7f95cbed STEP 4: RUN microdnf install --enablerepo=rhel-8-for-x86_64-baseos-beta-rpms traceroute Downloading metadata... Downloading metadata... Package Repository Size Installing: traceroute-3:2.1.0-6.el8.x86_64 rhel-8-for-x86_ 69.1 kB Transaction Summary: Installing: 1 packages Reinstalling: 0 packages Upgrading: 0 packages Removing: 0 packages Downgrading: 0 packages Downloading packages... Running transaction test... Installing: traceroute;3:2.1.0-6.el8;x86_64;rhel-8-for-x86_64-baseos-beta-rpms Complete. ERRO[0013] Can't add file /var/lib/containers/storage/overlay/75c2363d3c2166b78a97214c26ba7b002b588392ce92204956155c2d5addb385/diff/var/cache/yum/metadata/rhel-8-for-x86_64-appstream-beta-rpms-8-x86_64/gpgdir/S.gpg-agent to tar: archive/tar: sockets not supported ERRO[0013] Can't add file /var/lib/containers/storage/overlay/75c2363d3c2166b78a97214c26ba7b002b588392ce92204956155c2d5addb385/diff/var/cache/yum/metadata/rhel-8-for-x86_64-appstream-beta-rpms-8-x86_64/gpgdir/S.gpg-agent.browser to tar: archive/tar: sockets not supported ERRO[0013] Can't add file /var/lib/containers/storage/overlay/75c2363d3c2166b78a97214c26ba7b002b588392ce92204956155c2d5addb385/diff/var/cache/yum/metadata/rhel-8-for-x86_64-appstream-beta-rpms-8-x86_64/gpgdir/S.gpg-agent.extra to tar: archive/tar: sockets not supported ERRO[0013] Can't add file /var/lib/containers/storage/overlay/75c2363d3c2166b78a97214c26ba7b002b588392ce92204956155c2d5addb385/diff/var/cache/yum/metadata/rhel-8-for-x86_64-appstream-beta-rpms-8-x86_64/gpgdir/S.gpg-agent.ssh to tar: archive/tar: sockets not supported ERRO[0013] Can't add file /var/lib/containers/storage/overlay/75c2363d3c2166b78a97214c26ba7b002b588392ce92204956155c2d5addb385/diff/var/cache/yum/metadata/rhel-8-for-x86_64-baseos-beta-rpms-8-x86_64/gpgdir/S.gpg-agent to tar: archive/tar: sockets not supported ERRO[0013] Can't add file /var/lib/containers/storage/overlay/75c2363d3c2166b78a97214c26ba7b002b588392ce92204956155c2d5addb385/diff/var/cache/yum/metadata/rhel-8-for-x86_64-baseos-beta-rpms-8-x86_64/gpgdir/S.gpg-agent.browser to tar: archive/tar: sockets not supported ERRO[0013] Can't add file /var/lib/containers/storage/overlay/75c2363d3c2166b78a97214c26ba7b002b588392ce92204956155c2d5addb385/diff/var/cache/yum/metadata/rhel-8-for-x86_64-baseos-beta-rpms-8-x86_64/gpgdir/S.gpg-agent.extra to tar: archive/tar: sockets not supported ERRO[0013] Can't add file /var/lib/containers/storage/overlay/75c2363d3c2166b78a97214c26ba7b002b588392ce92204956155c2d5addb385/diff/var/cache/yum/metadata/rhel-8-for-x86_64-baseos-beta-rpms-8-x86_64/gpgdir/S.gpg-agent.ssh to tar: archive/tar: sockets not supported --> 6397680d1314255bae710ec7efdacb1b13b1e73394152aa5d3c32d6a2a85024a STEP 5: FROM 6397680d1314255bae710ec7efdacb1b13b1e73394152aa5d3c32d6a2a85024a STEP 6: CMD ["/bin/traceroute","google.com"] --> dc92a5c83e5ff85c81462203b7a264f8c12c09f5d9d58840cdfdcb16bff9d61b STEP 7: COMMIT traceroute [root@rhel8-beta traceroute]# podman images REPOSITORY TAG IMAGE ID CREATED SIZE localhost/traceroute latest dc92a5c83e5f 5 seconds ago 118MB localhost/rhel8-beta-hello latest 21f0cb513108 12 minutes ago 91.2MB localhost/rhel7-hello latest d4ab060a55e4 15 minutes ago 210MB docker.io/rhel8-beta/rhel latest a80dad1c1953 2 days ago 210MB registry.redhat.io/rhel8-beta/rhel latest a80dad1c1953 2 days ago 210MB registry.redhat.io/rhel8-beta/rhel-minimal latest 417409cdd8fe 2 days ago 91.2MB [root@rhel8-beta traceroute]#
Ok after talking and reviewing with Vivek we have found that this is not an issue with Overlay directly, the problem is with google golang tar, refuses to tar up socket files. So when the container image is created these socket files do not get added to the image. This is not really an issue. If Martin had properly change your run line to : RUN microdnf install --enablerepo=rhel-8-for-x86_64-baseos-beta-rpms traceroute; microdnf clean all It also goes away since the clean all removes the socket files.
Yes, adding 'microdnf clean all' fixes the issue. It has to be said though that it is not a proviso to run the 'microdnf clean' after installation of rpms although I have to say it makes sense within a container.
Should this be documented in the Beta container guide or rhel 8 Beta release notes?
(In reply to Laurie Friedman from comment #4) > Should this be documented in the Beta container guide or rhel 8 Beta release > notes? Laurie, Yeah, I think fixing this with thos docs is appropriate. Dan, Should we "patch" microdnf to do the "clean all" all the time? When would you NOT want to do this? Perhaps we have a switch, nocleanall? Best Regards Scott M
No microdnf should not be patched to do the clean all, this would cause secondary running of microdnf to pull down the cache again. Which could slow down builds. microdnf should probably clean up these sockets when it exits though, which would also fix the problem. Or it could put the sockets into /dev/shm (Only place inside of a container that is guaranteed to be a tmpfs.) microdnf on fedora does not exhibit this behaviour, not sure what is the difference between the versions.
> microdnf should probably clean up these sockets when it exits though I will look at it. I am sorry I don't have experience with podman. But what is the reason for storing the cache data into the image? You get bigger image. You need more time for creating tar. In general all application should work without content of /var/cache/. Do You have usage for old cache data later? If not, "microdnf clean all" is solution.
Jaroslav, yes you are correct as was pointed out earlier. The issue is, this can happen to a normal user as it did here. And we end up with bad looking warning/error messages. We don't see these if we use dnf/yum or if we use microdnf on Fedora. So I believe this is a bug. Easiest thing for you would be to clean these up when microdnf is done with them, I believe. Or to move them to a location that is sure to be ignored. /dev/shm sadly is the only place within a container to be sure. The container engines can not ignore /var/cache, since they have no idea whether or not users want this content.
The issue has a workaround: run microdnf clean all This is also a best practice when creating containers using DNF. Users don't generally want the cache which gets obsolete couple hours after creating the image. We'll keep this bug open and move it to 8.1.
Reprioritized and moved to 8.2 as there's an existing workaround. If it is a must have for 8.1 for any reason, please reach out to us.
The "this doesn't happen in Fedora part bothered me" greatly. I tried it again with the fedora-minimal image and it does recreate there as well (I used procps-ng instead of traceroute, but whatever): [jwboyer@zod traceroute]$ sudo podman build --rm --no-cache --force-rm -t procps-ng . STEP 1: FROM fedora-minimal STEP 2: MAINTAINER Martin Jenner "mjenner" --> ba77f659ee7e154b2ee3cda178625e01f65392acdab2b5353112cdbf92c0a029 STEP 3: FROM ba77f659ee7e154b2ee3cda178625e01f65392acdab2b5353112cdbf92c0a029 STEP 4: RUN microdnf install procps-ng Downloading metadata... Downloading metadata... Downloading metadata... Downloading metadata... Package Repository Size Installing: procps-ng-3.3.15-4.fc29.x86_64 fedora 328.6 kB Transaction Summary: Installing: 1 packages Reinstalling: 0 packages Upgrading: 0 packages Removing: 0 packages Downgrading: 0 packages Downloading packages... Running transaction test... Installing: procps-ng;3.3.15-4.fc29;x86_64;fedora Complete. ERRO[0184] Can't add file /var/lib/containers/storage/overlay/c155b4cd26209d3397b9155a8685998b8cc5efda34544724ad5e462fcb4ad199/diff/var/cache/yum/metadata/fedora-29-x86_64/gpgdir/S.gpg-agent to tar: archive/tar: sockets not supported ERRO[0184] Can't add file /var/lib/containers/storage/overlay/c155b4cd26209d3397b9155a8685998b8cc5efda34544724ad5e462fcb4ad199/diff/var/cache/yum/metadata/fedora-29-x86_64/gpgdir/S.gpg-agent.browser to tar: archive/tar: sockets not supported ERRO[0184] Can't add file /var/lib/containers/storage/overlay/c155b4cd26209d3397b9155a8685998b8cc5efda34544724ad5e462fcb4ad199/diff/var/cache/yum/metadata/fedora-29-x86_64/gpgdir/S.gpg-agent.extra to tar: archive/tar: sockets not supported ERRO[0184] Can't add file /var/lib/containers/storage/overlay/c155b4cd26209d3397b9155a8685998b8cc5efda34544724ad5e462fcb4ad199/diff/var/cache/yum/metadata/fedora-29-x86_64/gpgdir/S.gpg-agent.ssh to tar: archive/tar: sockets not supported ERRO[0184] Can't add file /var/lib/containers/storage/overlay/c155b4cd26209d3397b9155a8685998b8cc5efda34544724ad5e462fcb4ad199/diff/var/cache/yum/metadata/fedora-modular-29-x86_64/gpgdir/S.gpg-agent to tar: archive/tar: sockets not supported ERRO[0184] Can't add file /var/lib/containers/storage/overlay/c155b4cd26209d3397b9155a8685998b8cc5efda34544724ad5e462fcb4ad199/diff/var/cache/yum/metadata/fedora-modular-29-x86_64/gpgdir/S.gpg-agent.browser to tar: archive/tar: sockets not supported ERRO[0184] Can't add file /var/lib/containers/storage/overlay/c155b4cd26209d3397b9155a8685998b8cc5efda34544724ad5e462fcb4ad199/diff/var/cache/yum/metadata/fedora-modular-29-x86_64/gpgdir/S.gpg-agent.extra to tar: archive/tar: sockets not supported ERRO[0184] Can't add file /var/lib/containers/storage/overlay/c155b4cd26209d3397b9155a8685998b8cc5efda34544724ad5e462fcb4ad199/diff/var/cache/yum/metadata/fedora-modular-29-x86_64/gpgdir/S.gpg-agent.ssh to tar: archive/tar: sockets not supported ERRO[0184] Can't add file /var/lib/containers/storage/overlay/c155b4cd26209d3397b9155a8685998b8cc5efda34544724ad5e462fcb4ad199/diff/var/cache/yum/metadata/updates-29-x86_64/gpgdir/S.gpg-agent to tar: archive/tar: sockets not supported ERRO[0184] Can't add file /var/lib/containers/storage/overlay/c155b4cd26209d3397b9155a8685998b8cc5efda34544724ad5e462fcb4ad199/diff/var/cache/yum/metadata/updates-29-x86_64/gpgdir/S.gpg-agent.browser to tar: archive/tar: sockets not supported ERRO[0184] Can't add file /var/lib/containers/storage/overlay/c155b4cd26209d3397b9155a8685998b8cc5efda34544724ad5e462fcb4ad199/diff/var/cache/yum/metadata/updates-29-x86_64/gpgdir/S.gpg-agent.extra to tar: archive/tar: sockets not supported ERRO[0184] Can't add file /var/lib/containers/storage/overlay/c155b4cd26209d3397b9155a8685998b8cc5efda34544724ad5e462fcb4ad199/diff/var/cache/yum/metadata/updates-29-x86_64/gpgdir/S.gpg-agent.ssh to tar: archive/tar: sockets not supported ERRO[0185] Can't add file /var/lib/containers/storage/overlay/c155b4cd26209d3397b9155a8685998b8cc5efda34544724ad5e462fcb4ad199/diff/var/cache/yum/metadata/updates-modular-29-x86_64/gpgdir/S.gpg-agent to tar: archive/tar: sockets not supported ERRO[0185] Can't add file /var/lib/containers/storage/overlay/c155b4cd26209d3397b9155a8685998b8cc5efda34544724ad5e462fcb4ad199/diff/var/cache/yum/metadata/updates-modular-29-x86_64/gpgdir/S.gpg-agent.browser to tar: archive/tar: sockets not supported ERRO[0185] Can't add file /var/lib/containers/storage/overlay/c155b4cd26209d3397b9155a8685998b8cc5efda34544724ad5e462fcb4ad199/diff/var/cache/yum/metadata/updates-modular-29-x86_64/gpgdir/S.gpg-agent.extra to tar: archive/tar: sockets not supported ERRO[0185] Can't add file /var/lib/containers/storage/overlay/c155b4cd26209d3397b9155a8685998b8cc5efda34544724ad5e462fcb4ad199/diff/var/cache/yum/metadata/updates-modular-29-x86_64/gpgdir/S.gpg-agent.ssh to tar: archive/tar: sockets not supported --> f884083151adf6afb980404eaf8ba685d03cce0a6fa8931f91c31edc1028920c STEP 5: FROM f884083151adf6afb980404eaf8ba685d03cce0a6fa8931f91c31edc1028920c STEP 6: CMD ["/bin/traceroute","google.com"] --> c312136bbcb4340c9a6102bfae07426b7929e3883f1d130d65e8b814c80eb18d STEP 7: COMMIT procps-ng [jwboyer@zod traceroute]$ These socket files get created because gpg-agent is started during the transaction and specifies --homedir /var/cache/yum/metadata/<repoid>/gpgdir/. The agent itself exits, but the files aren't cleaned up.
I created patch that configures gpg-agent to not create/use sockets during repo key import. PR: https://github.com/rpm-software-management/librepo/pull/152
> Proposed gating test scenario: > - verify that there is no gpgdir under /var/cache/yum/metadata/ after installing a package using microdnf on a registered system. Karel, no. The bug is about sockets. We do not want to delete gpgdir directory (cached gpg keys). We want to verify that there are not sockets under gpgdir directory.
Karel, yes it is correct from my point of view.
The bug was reported on microdnf. But in fact the sockets are created by other applications (PackageKit, DNF, ...) too. The PR: https://github.com/rpm-software-management/librepo/pull/152 modifies librepo. Librepo does not create sockets during importing repo keys now. The modified code is used by context part of libdnf. So microdnf, PackageKit, ... are fixed now. I created patch for libdnf too. The PR https://github.com/rpm-software-management/libdnf/pull/731. The modified code is used by DNF. So DNF is fixed too.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2019:3583